# A Cryogenic Infrared Calibration Target

E.J. Wollack<EMAIL_ADDRESS>NASA Goddard Space Flight Center,
Greenbelt, MD 20771  R.E. Kinzer Jr  NASA Goddard Space Flight Center,
Greenbelt, MD 20771  S.A. Rinehart  NASA Goddard Space Flight Center,
Greenbelt, MD 20771

###### Abstract

A compact cryogenic calibration target is presented that has a peak diffuse
reflectance, $R\leq 0.003$, from $800-4,800\,{\rm cm}^{-1}$ $(12-2\,\mu$m).
Upon expanding the spectral range under consideration to $400-10,000\,{\rm
cm}^{-1}$ $(25-1\,\mu$m) the observed performance gracefully degrades to
$R\leq 0.02$ at the band edges. In the implementation described, a high-
thermal-conductivity metallic substrate is textured with a pyramidal tiling
and subsequently coated with a thin lossy dielectric coating that enables high
absorption and thermal uniformity across the target. The resulting target
assembly is lightweight, has a low-geometric profile, and has survived
repeated thermal cycling from room temperature to $\sim 4\,$K. Basic design
considerations, governing equations, and test data for realizing the structure
described are provided. The optical properties of selected absorptive
materials – Acktar Fractal Black, Aeroglaze Z306, and Stycast 2850 FT epoxy
loaded with stainless steel powder – are characterized and presented.

Lossy Dielectric Coatings; Calibration Targets; Infrared Instrumentation and
Techniques

> ##  I Introduction

Absorptive targets and light traps find widespread use in flux calibration,
termination of residual reflections in optical systems, and establishing the
zero point in reflectance spectrometry. Palchetti, Bianchini, and Castagnoli
(2008) Approximating a near-ideal absorber in a finite volume presents a
challenge for such applications – the absence of reflected light or how
“black” an object appears is not a unique material property or treatable as a
boundary condition within the framework of electromagnetic theory Sommerfeld
(1950) – but an object’s reflectance and absorptance are intimately tied to
the surface’s material properties and the underlying geometry of the
structure. de Hultst (1957) Thus to implement a low reflectance calibrator
design, the absorber material’s dielectric and magnetic properties need to be
either known or experimentally determined in order to suitably tailor the
target’s geometry.

At microwave through sub-millimeter wavelengths, precision absorptive
standards find use in radiometric flux calibration and low reflectance targets
for remote sensing, Gaidis, Anderson, and Harding (1999) astrophysics, Gush
and Halpern (1992); Mather  _et al._ (1999) and other metrology applications.
Janz, Boyd, and Ellis (1987) Elements of the design techniques used in these
microwave structures can be applied to realize compact, light-weight, and low-
reflectance absorbers while maintaining the overall manufacturability for far
infrared use at cryogenic temperatures. In this work, a calibration target for
cryogenic applications is explored and its performance is described in detail.
The design, manufacture and optical characterization of the target structure
and selected coatings of potential interest are summarized in Sections II,
III, and IV respectively.

##  II Target Design Considerations

Low reflectance targets have been realized from individual geometric
structures Gush and Halpern (1992); Mather  _et al._ (1999) or tilings Fixsen
_et al._ (2006) of absorbing dielectric cones or pyramids. Wedges, rulings
Gaidis, Anderson, and Harding (1999); Janz, Boyd, and Ellis (1987); Kogut  _et
al._ (2004) and Brewster angle light traps Elterman (1977); Breneman (1981);
Clarke and Compton (1986); LaRocca (1985) have also been employed in
instrumentation settings where only a single polarization state is of
interest. Through the microwave waveband, the absorptive coating’s finish is
typically smooth compared to the freespace wavelength, $\lambda_{o}$, and the
resulting reflections from surfaces making up the target’s facets are largely
specular. In this limit, the effective reflectance for a pyramidal wedge with
vertex angle, $\alpha$, can be estimated from geometric optics considerations,
$R\approx R_{o}^{\pi/\alpha}$. Here the single bounce reflection at the angle
of incidence, $R_{o}$, is estimated from the Fresnel coefficients. Carli
(1974) In effect, the geometry can be used to limit the impact of imperfect a
priori knowledge of the coating properties on the final target absorptance at
cryogenic temperatures. Wollack  _et al._ (2007) As a result, the key
geometric parameters that control the reflectance in this limit are the
details of the pyramid’s points, grooves at the structure base, and the number
of reflections within the structure. Fixsen  _et al._ (2006) Ideally these
features would be perfectly sharp; however, in practice a finite length scale
is achievable and set by fabrication tolerances and the coating morphology.

In the limit where the freespace wavelength, $\lambda_{o}$, is less than twice
the tiling period, $p$, the target’s response can be computed by treating the
structure as an artificial dielectric adiabatic impedance taper. Meyer and
Severin (1956); Kuester and Holloway (1994) As the wavelength is further
increased beyond the length scale of the taper, $h$, the target’s surface
appears smooth and incident fields evanescently propagate into the structure.
Baekelandt, Olyslager, and De Zutter (1999) As a result, the radiation is
largely specularly reflected from the structure. More generally, these
arguments can be extended to treat the diffuse reflectance from the surface
roughness in the design process if sufficient details are available. Bedford
and Ma (1974) Here the experimentally observed diffuse reflectance is taken as
a property of the coating that in turn determines acceptable substrate
geometry and the short wavelength limit of the structure’s response.

The use of an appropriate coating with a controlled and sufficient thickness
to absorb the incident radiation is also an important consideration. Here the
material is assumed to be non-magnetic and characterized as a function of
frequency by a complex dielectric function,
$\epsilon_{r}=\epsilon_{r}^{\prime}+i\cdot\epsilon_{r}^{\prime\prime}$,
relative to the permittivity of freespace. The thickness of the target’s lossy
dielectric layer employed must be greater than the material’s field
penetration length scale,
$\delta=\lambda_{o}/\pi\sqrt{2\epsilon_{r}^{\prime\prime}}$, to suitably
attenuate signals before reflection from the underlying substrate. Wollack
_et al._ (2007) By design, the substrate is optically thick and the
absorptance is related to the reflectance by, $A=1-R$. Consideration of the
Fresnel coefficient magnitude at the surface of the lossy dielectric reveals
that a lower index coating leads to lower insertion loss per bounce and thus
has greater absorptance from the overall structure in the long wavelength
limit. In the infrared, a reduction in coating thickness is desirable as long
as the coating maintains structural integrity and minimal thermal gradients.

With appropriate attention to detail a variety of coatings could potentially
be employed in this application Smith (1984); Betts  _et al._ (1985); Persky
(1999) to achieve the desired response. The service temperature of epoxies,
paints, and deposited films are typically limited by the material’s adhesion
and thermal match to the substrate material over the temperature range of
interest. For high temperature applications, chemical breakdown and aging may
also require more careful consideration. At cryogenic temperatures typical
anodized coating (thermal conductivity, $\kappa\sim 1\,$W/m/K) with thickness,
$\delta x<100\,\mu$m, produces a thermal gradient across the absorber, $\Delta
T/q\simeq\delta x/\kappa\sim 0.1\,$K/mW per unit incident flux, $q$. For our
application this consideration is subdominant to the $\sim 0.2\,$K thermometry
calibration uncertainty at 4 K. Cooling or achieving precise knowledge of the
target temperature becomes a greater concern in going from bulk to low-volume-
filling-fraction metal or semiconducting foams Advena, Bly, and Cox (1993);
Becker, Fettig, and Ruppel (1999) and semi-metallic nanotubes Quijada  _et
al._ (2011) in the long wavelength limit due to such coatings’ lower thermal
conductivity. Additional considerations for space qualification and other low-
outgassing applications can be found in the literature. Kauder (2006); Kralik
and Katsir (2009); Salomon  _et al._ (2009) Based upon prior experience for
the target implementation described here we use a custom specified thickness
of Acktar Fractal Black Ack  (Acktar FB) amenable to a variation on the
vendor’s standard ($11\,\mu$m thickness coating) process for the absorptive
coating.

Figure 1:  The shaded gray regions in each figure represent the high thermal
conductivity and optically thick patterned metallic substrate. The solid black
lines are the lossy dielectric layer with thickness greater than the
electrical penetration length, $\delta$. Representative geometric ray traces
are indicated with dashed lines. The minimal number of bounces occur when rays
go through the focus formed by the slot’s radius (black dots, right). For a
simple slot, the single bounce observation angle is controlled by the slot
width and depth (shaded region, upper right). The overall absorptance can be
improved through the addition of a “dog leg” in the slot as shown (lower
right).

##  III  Sample Preparation and Fabrication

In implementing the absorptive calibrator design we concentrate on the use of
simple manufacturing approaches to achieve the desired response in the
infrared. The underlying approach adopted was to use a lossy dielectric on a
highly conductive substrate to achieve high absorptance. See Figure 1 for
target geometry and ray-trace summary. An aluminum alloy, 6061-T6, is chosen
as the target’s substrate material. It is durable, easy to machine, and its
high thermal conductance minimizes thermal gradients and enables knowledge of
the device temperature. The design described is driven by a need for the
target to have a total thickness $<6\,$mm. A total calibrator diameter,
$d_{\rm cal}=25\,$mm, was chosen for compatibility with our existing
integrating sphere configuration and for use as a low-reflectance reference in
a cryogenic filter wheel assembly. A pyramidal tiling of the surface is used
to limit the total required thickness of the target relative to a single
pyramid.

The tips of the pyramids (radius $<20\,\mu$m) used in the tiling were readily
achievable by wire electrostatic discharge milling (EDM) with a full apex
angle, $\alpha=\pi/6$, and are comparable to the surface roughness features of
the coatings employed. A finite width slot was substituted for the sharp
groove found at the base of the pyramid in a traditional microwave target
design. Slots $s\simeq 250\,\mu$m in width, $d=940\,\mu$m deep, on a pitch
$p=1370\,\mu$m were defined by the EDM wire diameter. The finite slot width
relative to the pitch limits the achievable reduction in the pyramidal tiling
thickness by a factor of $2d_{\rm cal}h/p^{2}\sim 50$. The resulting patterned
features in the aluminum substrate are less than half of the component’s
thickness and the remainder of the substrate is used to provide low lateral
thermal conduction, mounting points for cooling, and thermometry.

A cross-section of the fabricated pyramidal target’s pyramidal tiles are shown
in the upper panel of Figure 1. In the specular limit, this geometry has $>6$
reflections for $80\%$ of rays incident on the pyramidal structure. The ratio
of the slot width over the depth limits number of rays with a single bounce
and sets the target reflectance in this structure. By tipping the target to
$\sim 10^{\circ}$ from the normal the specular reflectance contribution from
the bottom of the slot can be reduced. Alternatively, a “dog leg” at the
bottom of the channel as shown in the lower panel of Figure 1 can be used to
improve the absorptance by increasing the number of bounces ($N\geq 3$) in the
slot region and effectively reduces the angular extent of the caustic arising
from the slot’s finite radius. If greater volume for the target is available,
the influence of the finite slot width can be further reduced by using an
adiabatically curved path for the slot to geometrically hide its termination
from view.

The pyramidally tiled absorber structures and optical witness flats were
subsequently coated with Acktar FB Ack  via Physical Vapor Deposition (PVD)
for spectral characterization. Two coating thicknesses, $\sim 25\,\mu$m and
$\sim 50\,\mu$m, were specified in order to insure sufficient coating
absorption. See Figure 2 for detailed digital images and a summary of the
samples optically characterized in this study.

Figure 2: Images of substrate samples: (top-left) $300\,\mu$m Steelcast
(Stycast 2850 FT epoxy loaded with $30\%$ stainless steel powder by volume)
sample on aluminum flat; (top-center) $40\,\mu$m Aeroglaze Z306 black paint on
Steelcast sample for enhanced long and short wavelength response; (top-right)
Detail of Acktar FB coated calibrator on a substrate roughened with a
pyramidal tiling viewed approximately $\pi/4$ from vertical; (middle-left)
$25\,\mu$m Acktar FB witness coating on aluminum flat; (middle-center)
$25\,\mu$m Acktar FB calibrator roughened with a pyramidal tiling; (middle-
right) SEM image of $25\,\mu$m Acktar FB coating; (lower-left) $56\,\mu$m
Acktar FB witness coating on aluminum flat; (lower-center) $56\,\mu$m Acktar
FB calibrator roughened with a pyramidal tiling; and (lower-right) SEM image
of $56\,\mu$m Acktar FB coating. The images of the “flat” and “rough” samples
(i.e., left- and center-columns) were taken as a single exposure under an
intense visible light source from approximately normal incidence. The
logarithmic brightness scale has been uniformly adjusted in all digital images
of the samples to convey the full range of the intensity recorded. The grey
scale in the scanning electron microscope images (i.e., middle- and lower-
right panels of figures) are uncorrected and convey the typical morphology of
the Acktar FB samples investigated.

In addition, a flat sample was coated with “Steelcast”, a dielectric mixture
comprised of Emerson Cumming 2850 FT epoxy with a $30\%$ volume filing
fraction of $8\,\mu$m stainless steel powder. Wollack  _et al._ (2008) The
sample was then hand lapped to $\sim 300\,\mu$m thickness with 600 grit sand
paper. The relative dielectric permittivity and magnetic permeability of this
dielectric mixture are $\epsilon_{r}\approx 10.6+2.1i$ and $\mu_{r}\approx 1$,
respectively. These properties were verified for the mixture employed by using
a witness coupon as a waveguide Fabry-Perot resonator Wollack  _et al._ (2008)
at $1\,{\rm cm^{-1}}$. To explore improving the Steelcast low frequency
reflectance, a sample was coated with an anti-reflection layer realized from a
$\sim 40\,\mu$m thick layer of Aeroglaze Z306 paint Aer  with an RMS surface
roughness, $\sigma\simeq 2\,\mu$m. In modeling this multi-layer dielectric
coating Yeh (1988); Smith (1984) an effective relative permittivity,
$\epsilon_{r}\approx 2.6+0.6i$, is estimated for the paint on the epoxy
substrate using the measured sample parameters and the specular reflectance
$<500\,{\rm cm^{-1}}$ (e.g., see the spectrum in Fig. 3 labeled “AG:Z306”).

This combination of materials finds use in targets and baffles in cryogenic
applications below $\sim 400\,{\rm cm^{-1}}$ and provides an informative point
of reference to the microwave structures motivating the calibrator design
presented. Loaded epoxy structures are typically fabricated by building up the
absorber in layers via spraying, molding, or with great difficultly direct
machining on a textured substrate. The achievable tolerances with these
approaches have higher variability than is observed with PVD coating on an
identical patterned substrate in the design’s critical tip and slot regions.
As a result, a significantly greater calibrator volume is required to insure
identical optical performance in practice and the structure’s thermal time
constant and gradients are correspondingly increased. While relatively high
index loaded epoxies provide a path in the microwave to minimize the coating
thickness and mass, in going to the infrared, lower index lossy thin films and
conversion coatings more naturally serve this purpose. Realizing the desired
substrate geometry presents the primary challenge to be address in this path
to calibrator volume reduction.

##  IV FTS Characterization

Infrared and sub-millimeter spectra were obtained using a Bruker IFS125hr
Fourier transform spectrometer (FTS). This instrument is equipped with
internal DTGS (Deuterated Triglycine Sulfate) detectors for use in the near-
through far-infrared and an external LHe-cooled bolometer with two
interchangeable filters for far-infrared and sub-millimeter wavelengths
($100-700$ and $5-100\,{\rm cm}^{-1}$, respectively). An array of
beamsplitters is used for the different spectral regions; the following were
used here: Si on CaF2 (near-IR), Ge on KBr (mid-IR), $6\,\mu$m multi-layer
Mylar (Bruker T222/2), and $75\,\mu$m Mylar (far-infrared and sub-millimeter).
The FTS is equipped with three internal light sources: tungsten (near-IR), SiC
Globar (mid-IR), and a mercury arc lamp (far-IR). Reflectance spectra for the
samples were measured in each band and then merged together to form a
composite spectrum. Spectra are recorded in vacuo ($\sim 0.3\,$mbar).

Figure 3: Measured specular reflectance. Dashed lines denote the stainless
steel loaded 2850 FT epoxy (gray) and the Aeroglaze Z306 paint (black); solid
lines are the Acktar FB flat $25\,\mu$m (gray) and $56\,\mu$m (black)
coatings; and the bold lines are the Acktar FB $25\,\mu$m (gray) and
$56\,\mu$m (black) coatings on substrates roughened with a pyramidal tiling.
The coatings are deposited on aluminum substrates.  Figure 4: Measured diffuse
reflectance. Dashed lines denote the stainless steel loaded 2850 FT epoxy
(gray) and the Aeroglaze Z306 paint (black); solid lines are the Acktar FB
flat $25\,\mu$m (gray) and $56\,\mu$m (black) coatings; and the bold lines are
the Acktar FB $25\,\mu$m (gray) and $56\,\mu$m (black) coatings on substrates
roughened with a pyramidal tiling. All coatings are deposited on aluminum
substrate disks. The room temperature Bruker integrating sphere (300 K,
$>250\,{\rm cm}^{-1}$) and the OPASI-T cryogenic reflectometer (4.2 K,
$<650\,{\rm cm}^{-1}$) data were calibrated with a polished aluminum
reference. The measured data from the two instrument configurations are in
agreement in the region of shared spectral overlap.

Specular reflectance data were recorded over the frequency range of
$20-5000\,{\rm cm}^{-1}$ at a $2\,{\rm cm}^{-1}$ resolution using a Pike
Technologies accessory (Part No. SKU2084). Light from the spectrometer takes a
“W”-path through the accessory, which has collimating optics and a fixed
$10^{\circ}$ angle of incidence on the sample. In this set-up, the sample’s
reflectance spectrum is obtained by taking the ratio of the light intensity
detected from the sample being measured to the intensity of an gold coated
flat provided with the Pike accessory. See Figure 3 for the observed specular
reflectance spectra.

Diffuse reflectance spectra were recorded using a Bruker integrating sphere
accessory (Part No. A562G/Q) over a frequency range of $250-10000\,{\rm
cm}^{-1}$. This accessory is equipped with a side-mounted DTGS detector used
primarily for mid- and near- infrared wavelengths and is supplied with a
diffusely reflecting gold reference. Internal absorptive baffles are used to
shield the detector from receiving light reflected directly from the samples.
In order to mitigate the typical noise of measurements with the integrating
sphere apparatus, and to help enhance the broad structure of the features
(e.g., see the Acktar FB $25\,\mu$m flat coating between $\sim 1000-3000\,{\rm
cm}^{-1}$), spectra were recorded using spectral resolutions of 8 and
$16\,{\rm cm}^{-1}$. In addition to the Bruker integrating sphere accessory,
diffuse reflectance was measured for the roughened Acktar FB $56\,\mu$m
coating using a custom cryogenic integrating sphere. Rinehart  _et al._ (2011)
This apparatus is equipped with a Si bolometer element (4.2 K) and an internal
temperature-controlled sample wheel. Far infrared spectra between
$100-650\,{\rm cm}^{-1}$ with $8\,{\rm cm}^{-1}$ resolution were measured
using this apparatus mounted to a Bruker IFS 113v FTS equipped with an
internal SiC Globar source and 3.5 and $12\,\mu$m Mylar beamsplitters. See
Figure 4 for observed diffuse reflectance spectra.

##  V Discussion

From the observed diffuse reflectance of the witness samples in Figure 4 one
notes both Acktar FB coatings have a slightly lower reflection than Aeroglaze
Z306 paint in the spectral range of interest. The calibrator constructed with
$56\,\mu$m of Acktar FB was noted to have the best overall optical performance
of the structures tested with an observed diffuse reflectance of $R<0.003$
between $800-4,800\,{\rm cm}^{-1}$ ($12-2\,\mu$m). The minimum in the diffuse
reflectance, $R\simeq 0.002$, occurs at $2000\,{\rm cm}^{-1}$ ($5\,\mu$m).
This represents a reduction in the diffuse reflectance by a factor of $\sim
50$ over a simple flat with an identical coating.

The low frequency response of the calibrator samples is set by the pyramidal
tiling’s pitch and finite coating thickness. At greater than $\sim 10^{\circ}$
from normal incidence the grey lines seen in Figure 2 that originate from
spectral reflectance off the bottom of the calibrator’s slots are no longer
discernible. This artifact can be further mitigated by gradually turning the
slot and removing its endpoint from view as indicated in Figure 1.
Incorporation of this fabrication detail is recommended to improve the overall
emittance. The calibrator is broadband, mechanically robust, and has survived
multiple cool downs to $\sim 4\,$K. Due to the structure’s low reflectance it
has been adopted as the preferred absorptive light trap for general laboratory
use.

## Acknowledgements

The authors gratefully acknowledge financial support from the NASA ROSES/APRA
program and thank E. Sharp and C. Wheeler for their contributions to
preparation and fabrication of the structures presented here. R. Kinzer was
supported by an appointment to the NASA Postdoctoral Program at GSFC,
administered by the Oak Ridge Associated Universities.

## References

  * Palchetti, Bianchini, and Castagnoli (2008) L. Palchetti, G. Bianchini, and F. Castagnoli, “Design and characterization of black-body sources for infrared wide-band Fourier transform spectroscopy,” Infrared Physics & Technology 51, 207–215 (2008).
  * Sommerfeld (1950) A. Sommerfeld,  _Optik – Vorlesungen $\ddot{u}$ber Theoretische Physik Band 4_ (Dieterichsche Verlagsbuchhandlung, Munich, 1950) pp. 265–266.
  * de Hultst (1957) H. V. de Hultst,  _Light Scattering by Small Particles_ (John Wiley and Sons, New York, 1957) p. 183.
  * Gaidis, Anderson, and Harding (1999) M. C. Gaidis, M. S. Anderson, and D. G. Harding, “Calibration target for far-infrared spaceborne applications,” Proc. SPIE 3795, 348–356 (1999).
  * Gush and Halpern (1992) H. P. Gush and M. Halpern, “Cooled submillimeter Fourier transform spectrometer flown on a rocket,” Review of Scientific Instruments 63, 3249–3260 (1992).
  * Mather  _et al._ (1999) J. C. Mather, D. J. Fixsen, R. A. Shafer, C. Mosier, and D. T. Wilkinson, “Calibrator Design for the COBE Far-Infrared Absolute Spectrophotometer (FIRAS),” ApJ 512, 511–520 (1999).
  * Janz, Boyd, and Ellis (1987) S. Janz, D. A. Boyd, and R. F. Ellis, “Reflectance characteristics in the submillimeter and millimeter wavelength region of a vacuum compatible absorber,” International Journal of Infrared and Millimeter Waves 8, 627–635 (1987).
  * Fixsen  _et al._ (2006) D. J. Fixsen, E. J. Wollack, A. Kogut, M. Limon, P. Mirel, J. Singal, and S. M. Fixsen, “Compact radiometric microwave calibrator,” Review of Scientific Instruments 77, 064905 (2006).
  * Kogut  _et al._ (2004) A. Kogut, E. Wollack, D. J. Fixsen, M. Limon, P. Mirel, S. Levin, M. Seiffert, and P. M. Lubin, “Design and calibration of a cryogenic blackbody calibrator at centimeter wavelengths,” Review of Scientific Instruments 75, 5079–5083 (2004) .
  * Elterman (1977) P. B. Elterman, “Brewster angle light trap,” Appl. Opt. 16, 2352–2352 (1977).
  * Breneman (1981) E. J. Breneman, “Light trap of high performance and simple construction,” Appl. Opt. 20, 1118–1119 (1981).
  * Clarke and Compton (1986) F. J. J. Clarke and J. A. Compton, “Correction methods for integrating-sphere measurement of hemispherical reflectance,” Color Research & Application 11, 253–262 (1986).
  * LaRocca (1985) A. LaRocca,  _The Infrared Handbook_ , edited by W. L. Wolfe and G. J. Zissis (Infrared Information and Analysis Center, Environmental Research Institute University of Michigan, 1985) Chap. 2, pp. 2–9.
  * Carli (1974) B. Carli, “Design of a blackbody reference standard for submillimeter region,” IEEE Trans. Microwave Theory Tech. MTT-22, 1094–1099 (1974).
  * Wollack  _et al._ (2007) E. J. Wollack, D. J. Fixsen, A. Kogut, M. Limon, P. Mirel, and J. Singal, “Radiometric-Waveguide Calibrators,” IEEE T. Instrumentation and Measurement 56, 2073–2078 (2007).
  * Meyer and Severin (1956) E. Meyer and H. Severin, “Absorption devices for electromagnetic centimeter waves and their acoustic analogs,” Z. Angew. Phys. 8, 105 –114 (1956).
  * Kuester and Holloway (1994) E. F. Kuester and C. Holloway, “A low-frequency model for wedge or pyramid absorber arrays-I: theory,” Electromagnetic Compatibility, IEEE Transactions on 36, 300–306 (1994).
  * Baekelandt, Olyslager, and De Zutter (1999) B. Baekelandt, F. Olyslager, and D. De Zutter, “Low-frequency reflectivity approximation for two- and three-dimensional em absorbers,” Electromagnetic Compatibility, IEEE Transactions on 41, 354–360 (1999).
  * Bedford and Ma (1974) R. E. Bedford and C. K. Ma, “Emissivities of diffuse cavities: Isothermal and nonisothermal cones and cylinders,” J. Opt. Soc. Am. 64, 339–349 (1974).
  * Smith (1984) S. Smith, “Specular reflectance of optical-black coatings in the far infrared,” Appl. Opt. 23, 2311–2326 (1984).
  * Betts  _et al._ (1985) D. Betts, F. Clarke, L. Cox, and J. Larkin, “Infrared reflection properties of five types of black coating for radiometric detectors,” Journal of Physics E: Scientific Instruments 18, 689–696 (1985).
  * Persky (1999) M. Persky, “Review of black surfaces for space-borne infrared systems,” Review of Scientific Instruments 70, 2193–2217 (1999).
  * Advena, Bly, and Cox (1993) D. Advena, V. Bly, and J. Cox, “Deposition and characterization of far-infrared absorbing gold black films,” Appl. Opt. 32, 1136–1144 (1993).
  * Becker, Fettig, and Ruppel (1999) W. Becker, R. Fettig, and W. Ruppel, “Optical and electrical properties of black gold layers in the far infrared,” Infrared Physics & Technology 40, 431–445 (1999).
  * Quijada  _et al._ (2011) M. A. Quijada, J. G. Hagopian, S. Getty, R. E. Kinzer, Jr., and E. J. Wollack, “Hemispherical reflectance and emittance properties of carbon nanotubes coatings at infrared wavelengths,” in  _Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series_, Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 8150 (2011).
  * Kauder (2006) L. Kauder, “Spacecraft thermal control coatings references,” Tech. Rep. TP-2005-212792 (NASA Goddard Space Flight Center, 2006).
  * Kralik and Katsir (2009) T. Kralik and D. Katsir, “Black surfaces for infrared, aerospace, and cryogenic applications,” in  _Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series_, Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 7298 (2009).
  * Salomon  _et al._ (2009) Y. Salomon, N. Sternberg, I. Gouzman, G. Lempert, E. Grossman, D. Katsir, R. Cotostiano, and T. Minton, “Qualification of Acktar Black Coatings for Space Application,” in  _Proceedings of the International Symposium on Materials in a Space Environment, Aix-En-Provence, France_ (2009).
  * (29) Acktar Ltd., “Acktar Fractal Black”, Leshem St., Entrance A, P.O.B. 8643, Kiryat-Gat 82000.
  * Wollack  _et al._ (2008) E. J. Wollack, D. J. Fixsen, R. Henry, A. Kogut, M. Limon, and P. Mirel, “Electromagnetic and Thermal Properties of a Conductively Loaded Epoxy,” International Journal of Infrared and Millimeter Waves 29, 51–61 (2008).
  * (31) Lord Corporation Chemical Products, “Aeroglaze Z306 Flat Black Absorptive Polyurethane Low-Outgassing Paint”, 2000 West Grandview Blvd., P.O. Box 10038, Erie, PA 16514-0038.
  * Yeh (1988) P. Yeh,  _Optical Waves in Layered Media_ (John Wiley & Sons, New York, 1988) Chap. 5, pp. 102–117.
  * Rinehart  _et al._ (2011) S. A. Rinehart, D. J. Benford, G. Cataldo, E. Dwek, R. Henry, J. R. E. Kinzer, J. Nuth, R. Silverberg, C. Wheeler, and E. Wollack, “Measuring the optical properties of astrophysical dust analogues: instrumentation and methods,” Appl. Opt. 50, 4115–4123 (2011).


# Clickbait Spoiling via Question Answering and Passage Retrieval

Matthias Hagen1
Maik Fröbe1 Artur Jurk1
Martin Potthast2

1Martin-Luther-Universität Halle-Wittenberg
2Leipzig University

###### Abstract

We introduce and study the task of clickbait spoiling: generating a short text
that satisfies the curiosity induced by a clickbait post. Clickbait links to a
web page and advertises its contents by arousing curiosity instead of
providing an informative summary. Our contributions are approaches to classify
the type of spoiler needed (i.e., a phrase or a passage), and to generate
appropriate spoilers. A large-scale evaluation and error analysis on a new
corpus of 5,000 manually spoiled clickbait posts—the Webis Clickbait Spoiling
Corpus 2022—shows that our spoiler type classifier achieves an accuracy of
80%, while the question answering model DeBERTa-large outperforms all others
in generating spoilers for both types.

##  1 Introduction

Clickbait is the term used to describe posts in social media that are intended
to inappropriately entice their readers to visit a web page. This is achieved
through formulations such as sensationalism or cataphors that are believed to
create a so-called curiosity gap: “a form of cognitively induced deprivation
that arises from the perception of a gap in knowledge or understanding”
Loewenstein (1994). Clickbait is perceived as inappropriate since its
resolution is usually ordinary or trivial, comprising little more than a
phrase, short passage, or a list of things that could just as easily have been
included in the post. This observation motivates us to introduce the task of
clickbait spoiling: identifying or generating a spoiler for a clickbait post.

Figure 1 shows four examples of clickbait on Twitter, along with spoilers. The
first two tweets explicitly or implicitly promise a surprising resolution to
spark curiosity, but their spoilers are brief and trivial. The linked page of
the first tweet adds almost nothing, and the spoiler of the second is common
sense. The third spoiler is a passage from the linked page, and the fourth is
a list of things. Even though there are length limits to the informativeness
of tweets, the spoilers in all examples could easily have been part of the
original tweets.

Figure 1: Examples of clickbait tweets and spoilers for them extracted from
the respective linked web page.

This paper reports about our investigation into clickbait spoiling and the
following contributions: (1) The Webis Clickbait Spoiling Corpus 2022 (Webis-
Clickbait-22), consisting of 5,000 clickbait posts, their linked pages and a
spoiling piece of text therein.111Data: https://webis.de/data.html?q=clickbait
(2) A two-step approach to clickbait spoiling that first classifies a
clickbait post according to its spoiler type (phrase or passage), and then
treats spoiling either as a question answering or as a passage retrieval task.
(3) A systematic evaluation of state-of-the-art methods for spoiler type
classification, question answering, and passage retrieval.222Code:
https://github.com/webis-de/ACL-22 Although the first step of spoiler type
classification is not necessary, our results suggest that it can be helpful.
Even more so, as we have not yet tackled multipart spoilers (bottom example in
Figure 1; 876 cases also part of our corpus) that probably require a different
spoiling approach.

##  2 Related Work

Following an overview of research on clickbait and its operationalization so
far, models of question answering and passage retrieval are examined.

###  2.1 Clickbait and its Operationalization

The underlying assumption of most research on clickbait is that it is a form
of data-driven optimization of social media posts to exploit the curiosity gap
described by Loewenstein (1994). At least that’s what Peter Koechley (2012),
the CEO of Upworthy, claimed. Upworthy became one of the first major spreaders
of clickbait on Facebook, and their success has prompted Facebook to change
its news recommendation algorithms to curb the amount of clickbait, twice El-
Arini and Tang (2014); Peysakhovich and Hendrix (2016).

Exploratory and theoretical studies of clickbait and its impact on journalism
analyzed its prevalence for more than 150 publishers Rony et al. (2017); its
economics for the news market Munger (2020); its impact on perceptions of
credibility and quality (overall negative) Molyneux and Coddington (2020); and
noted a slow decline over the past decade Lischka and Garz (2021).

Journalistic studies of this kind rely on clickbait detection technologies.
Originally proposed by Rubin et al. (2015) but not followed up, Potthast et
al. (2016) and Chakraborty et al. (2016) independently developed the first
detectors. Starting from a shared task organized by Potthast et al. (2018)
shortly after, more than 50 approaches have been contributed to date. An
overview is beyond the scope of our work, but transformer models dominate this
task as well. For the clickbait generation task, preceded by a rule-based
generator Eidnes (2015), only Shu et al. (2018) and Xu et al. (2019) have
presented more advanced models, while Karn et al. (2019) generate teaser
headlines that are explicitly not meant to be clickbait. So far, no attempt
has been made to generate spoilers for clickbait.

###  2.2 Question Answering

If one considers clickbait spoiling as a question answering problem, there are
numerous possible solutions. Among the available question-answering benchmarks
Dzendzik et al. (2021), we select two to choose appropriate state-of-the-art
models for our evaluation: (1) SQuAD Rajpurkar et al. (2016) compiles 107,785
questions and answers based on 536 Wikipedia articles. Although a wide range
of questions and answers are included, the vast majority of 93.6% are factual
(32% names, 31.8% noun phrases, 19.8% numbers, 5.5% verb phrases, and 3.9%
adjective phrases), while the remainder are descriptive (3.7% clauses and 2.7%
other). We use SQuAD v1.1, not the v2.0 superset Rajpurkar et al. (2018),
which contains unanswerable questions, since we do not expect clickbait to be
“unspoilable”. (2) TriviaQA Joshi et al. (2017) contains 95,000
question–answer pairs, mostly dealing with trivia questions that are supposed
to be particularly difficult to answer. These are comparable to clickbait in
that many of them address rather trivial things (see Figure 1).

The question answering models used in our experiments are ALBERT Lan et al.
(2020), AllenAI-Document-QA Clark and Gardner (2018), BERT (cased/uncased)
Devlin et al. (2019), Big Bird Zaheer et al. (2020), DeBERTa (large) He et al.
(2021), ELECTRA Clark et al. (2020), FunnelTransformer Dai et al. (2020),
MPNet Song et al. (2020), and RoBERTa (base/large) Liu et al. (2019). Many of
them are or were state of the art on the above benchmarks and implement
various different architectural paradigms.

###  2.3 Passage Retrieval

Passage retrieval relaxes the question answering task a bit in the sense of
allowing longer passages of text as answers (e.g., one or more sentences),
rather than exact phrases or statements. Neural retrieval models, as surveyed
by Guo et al. (2020) and Lin et al. (2021), have been successfully applied to
passage retrieval. One of the most important passage retrieval benchmarks is
part of MS MARCO, a series of challenges whose first edition was a large
question answering task Nguyen et al. (2016). A passage retrieval dataset of
8.8 million passages was derived for the underlying set of 100,000 questions
originally submitted to Bing. This dataset formed the basis for two
consecutive shared tasks at the TREC 2019 and 2020 Deep Learning tracks
Craswell et al. (2019, 2020).

The passage retrieval models used in our experiments are MonoBERT Nogueira and
Cho (2019); Nogueira et al. (2019) and MonoT5 Nogueira et al. (2020) (both
topped the MS MARCO passage retrieval leaderboard once), and the classic
baseline models BM25 Robertson and Zaragoza (2009) and Query Likelihood Ponte
and Croft (1998), implemented in Anserini Yang et al. (2017).

##  3 Webis Clickbait Spoiling Corpus 2022

To tackle clickbait spoiling for the first time, we created the Webis
Clickbait Spoiling Corpus 2022 (Webis-Clickbait-22), a collection of 5,000
clickbait posts and their associated spoilers.

###  3.1 Corpus Construction

Our corpus is primarily based on five social media accounts on Twitter,
Reddit, and Facebook that manually spoil clickbait: r/savedyouaclick,
@HuffPoSpoilers, @SavedYouAClick, @UpworthySpoiler, and
@StopClickBaitOfficial. With the goal of collecting 5,000 “spoilable”
clickbait posts at an expected rejection rate of around 10% of unusable posts,
5,555 were initially collected from the accounts. Each of them was manually
reviewed, and those that turned out not to be spoiled clickbait were removed
(e.g., funny posts not intended to be spoilers, or posts with unavailable
linked documents). The rejection rate was higher than expected, and only 4,204
posts remained.

To reach our goal of 5,000 posts, we then sampled from the Webis-Clickbait-17
corpus used in the Clickbait Challenge 2017 Potthast et al. (2018). The corpus
contains 38,517 tweets, each of which was rated by 5 annotators on a 4-point
Likert scale for clickbaitiness: “no clickbait,” “slight clickbait,”
“considerable clickbait,” and “heavy clickbait.” Of the tweets, 1,845 scored
an average of 0.8 or higher and can safely be considered clickbait. We
selected tweets from this subset and manually spoiled them based on the linked
document until our target size of 5,000 posts was reached.

Table 1: Key statistics of the Webis Clickbait Spoiling Corpus 2022 (Webis-Clickbait-22). Source | Spoiler | Entries | Average text length  $\mathbf{\pm}$ Std.Dev. | Corpus splits | Top source
---|---|---|---|---|---
|  |  |  | Post | Document | Spoiler | Train | Val. | Test | Name | Count
| Phrase | 342 |  | 13.4  $\mathbf{\pm}$3.6 | 433.7  $\mathbf{\pm}$347.90 |  03.0  $\mathbf{\pm}$1.60 | 221 | 45 | 76 | Stop Clickbait | 342
Facebook | Passage | 388 |  | 13.4  $\mathbf{\pm}$4.0 | 490.9  $\mathbf{\pm}$351.50 | 24.9  $\mathbf{\pm}$20.0 | 231 | 73 | 84 | Stop Clickbait | 388
| Multipart | 94 |  | 14.2  $\mathbf{\pm}$4.1 | 651.8  $\mathbf{\pm}$545.20 | 28.5  $\mathbf{\pm}$33.0 | 68 | 12 | 14 | Stop Clickbait | 94
| Phrase | 688 |  | 13.2  $\mathbf{\pm}$4.0 | 584.6  $\mathbf{\pm}$798.60 |  02.8  $\mathbf{\pm}$1.60 | 455 | 109 | 124 | savedyouaclick | 688
Reddit | Passage | 859 |  | 13.1  $\mathbf{\pm}$4.0 | 657.2  $\mathbf{\pm}$1004.7 | 25.4  $\mathbf{\pm}$20.3 | 533 | 148 | 178 | savedyouaclick | 859
| Multipart | 250 |  | 12.8  $\mathbf{\pm}$4.4 | 991.7  $\mathbf{\pm}$899.50 | 32.7  $\mathbf{\pm}$36.2 | 162 | 46 | 42 | savedyouaclick | 250
| Phrase | 1,095 |  | 11.0  $\mathbf{\pm}$3.4 | 479.1  $\mathbf{\pm}$502.90 |  02.7  $\mathbf{\pm}$1.70 | 691 | 181 | 223 | HuffPoSpoilers | 794
Twitter | Passage | 752 |  | 10.3  $\mathbf{\pm}$4.2 | 597.4  $\mathbf{\pm}$605.80 | 22.3  $\mathbf{\pm}$ 13.5 | 510 | 101 | 141 | HuffPoSpoilers | 328
| Multipart | 532 |  | 11.5  $\mathbf{\pm}$3.8 | 884.0  $\mathbf{\pm}$930.30 | 35.4  $\mathbf{\pm}$34.4 | 329 | 85 | 118 | HuffPoSpoilers | 148
| Phrase | 2,125 |  | 12.1  $\mathbf{\pm}$3.8 | 505.9  $\mathbf{\pm}$599.4 | 2.8  $\mathbf{\pm}$1.6 | 1,367 | 335 | 423 | HuffPoSpoilers | 794
$\sum$ | Passage | 1,999 |  | 12.1  $\mathbf{\pm}$4.3 | 602.4  $\mathbf{\pm}$774.0 | 24.1  $\mathbf{\pm}$18.1 | 1,274 | 322 | 403 | savedyouaclick | 859
| Multipart | 876 |  | 12.2  $\mathbf{\pm}$4.1 | 889.8  $\mathbf{\pm}$892.2 | 33.9  $\mathbf{\pm}$34.8 | 559 | 143 | 174 | savedyouaclick | 250

Thus, our final corpus consists of 4,204 posts from Twitter, Reddit, and
Facebook that were spoiled by a third party specializing in this task, and 796
tweets from the Webis-Clickbait-17 corpus with an average clickbaitiness of at
least 0.8 that we spoiled ourselves. For each of the 5,000 clickbait posts, we
also reviewed and corrected erroneous spoilers and labeled their exact
positions in the linked documents. Our internal guidelines dictated that a
spoiler should be as short as possible (i.e., if one word is enough, not a
whole sentence should be chosen). Since the underlying annotation task is
simple, one main annotator was sufficient. Nevertheless, randomly selected as
well as ambiguous cases were discussed with two additional experts among the
co-authors. No systematic errors or unforeseen difficulties in solving the
annotation task were identified during these discussions.

During our annotation, we found that none of the common approaches to main
content extraction worked reliably for all the documents linked in the
clickbait posts. Yet, clean content is a prerequisite for research on
clickbait spoiling to eliminate as many confounding variables as possible. To
ensure a clean corpus, one annotator manually extracted the main content of
the linked documents, removing (inline) advertisements, links to related
articles (e.g., “READ ALSO: […]” or “Also from CNBC […]”), credits (e.g.,
“Image credit: […]” or “Photo by […]”), and social media links (e.g.,
“Subscribe to […]” or “Follow us on […]”). A random selection was reviewed to
ensure high quality.

Moreover, during spoiler annotation, it turned out that there are basically
three types of spoilers: (1)  _phrase spoilers_ consisting of a single word or
phrase from the linked document (e.g., the first two spoilers in Figure 1, but
often named entity spoilers as well), (2)  _passage spoilers_ consisting of
one or a few sentences of the linked document (e.g., the third spoiler in
Figure 1), and (3)  _multipart spoilers_ consisting of more than one non-
consecutive phrases or passages of the linked document (e.g., the fourth
spoiler in Figure 1). Spoiler types were also annotated by the main annotator,
and randomly checked by the other two.

In sum, each of the 5,000 posts in our corpus consists of a unique ID, the
platform from which it was taken, the respective platform’s post ID, the
post’s text (i.e., the “clickbait”), the URL to the linked document, the
manually extracted title and paragraph-divided main content of the linked
document, the manually optimized spoiler, the spoiler’s character position in
the main content, and the type of spoiler (phrase, passage, or multipart). In
total, the annotation took about 560 hours, which marked the limit of our
budget dedicated for this step.

###  3.2 Corpus Statistics

Table 1 summarizes the main statistics of our corpus. Most spoiled clickbait
posts come from Twitter (47.5%) and Reddit (36%), whereas the Facebook account
contributes less (16.5%). Most spoilers are phrases (42.5%) and passages
(40%). That there are fewer multi-part spoilers could be due to the fact that
spoiler account operators prefer to spoil “simpler” clickbait posts. For the
corpus, we also provide a fixed random 80/20/20 train/validation/test split to
ensure future reproducibility and comparability with our results.

##  4 Type-dependent Clickbait Spoiling

Our approach to clickbait spoiling is based on the observation that there are
three types of spoilers: (1) phrase spoilers, (2) passage spoilers, and (3)
multipart spoilers. We assume that different tailored approaches will work
best for each spoiler type. However, an important prerequisite for this is the
corresponding classification of clickbait. Therefore, we first investigate how
well the spoiler type of a clickbait post can be predicted (Section 4.1).

The generation of phrase and passage spoilers for a given clickbait post is
similar in that the solution to the problem in both cases amounts to
extracting a coherent piece of text from the linked document. To this end,
there are a variety of existing approaches in related disciplines whose output
is either a phrase or a passage, and which may be adapted to clickbait
spoiling. We therefore investigate whether phrase spoilers can be identified
by conventional question answering methods (i.e., we treat a clickbait post as
a “question” to which a phrase of the linked document should be returned as
the “answer”; Section 4.2), and whether passage spoilers can be identified by
conventional passage retrieval methods (i.e., we treat a clickbait post as a
“query” and the paragraphs of the linked document as the collection from which
to retrieve the best “passage”; Section 4.3). In our evaluation, we focus on
phrase and passage spoilers and also examine the abilities of the above
question answering and passage retrieval methods to serve as one-size-fits-all
solutions for phrases and passages. For multipart spoilers, a novel approach
will be needed, which is beyond the scope of our current work but an
interesting direction for the future.

###  4.1 Spoiler Type Classification

For the spoiler classification subtask, we experimented with classic feature-
based models (Naïve Bayes, Logistic Regression, SVM) and the neural models
BERT-, DeBERTa-, and RoBERTa.

As feature types for the classic models, we use $\mathit{tf}$\- and
$\mathit{tf}\kern-1.49994pt\cdot\kern-1.49994pt\mathit{idf}$-weighted word and
POS tag uni- and bigrams from the clickbait post and
$\mathit{tf}\kern-1.49994pt\cdot\kern-1.49994pt\mathit{idf}$-weighted word and
POS tag uni- and bigrams from the linked document. We include features from
the linked document, since it has to be analyzed for the spoiler generation
anyway. The $\mathit{idf}$values are calculated on the OpenWebText corpus
Gokaslan and Cohen (2019) to prevent any bias from the comparatively small
size of our corpus.

The input for the neural models is a post concatenated with the main content
of the linked document.

###  4.2 Phrase Spoiler Generation

Viewing a clickbait post for which a phrase spoiler should be derived as a
“question” and the linked document as potentially containing an “answer”,
phrase spoiler generation can be tackled by question answering methods. We
therefore employ ten state-of-the-art question answering methods trained on
the SQuAD data and fine-tune them on our new clickbait spoiling training set:
ALBERT, BERT (cased/uncased), BigBird, DeBERTa (large), ELECTRA,
FunnelTransformer, MPNet, and RoBERTa (base/large).

###  4.3 Passage Spoiler Generation

Treating the clickbait post whose spoiler type is a passage as a “query” for
which the “most relevant” passage from the linked document is to be retrieved,
passage spoiler generation can be tackled by passage retrieval methods. We
therefore use ten state-of-the-art passage retrieval approaches trained on the
MS MARCO data: BM25 and QLD in four variants each (alone or with RM3/Ax/PRF
query expansion), MonoBERT, and MonoT5. In addition, we also adapt all of the
above question answering models to retrieve passages by simply considering the
passage as the returned result from which the question answering model
extracts its answer.

##  5 Evaluation of Spoiler Type Classification

In our evaluation, we assume a setup in which a previous clickbait detection
would have (perfectly) identified posts as clickbait. To then evaluate the
effectiveness of spoiler type classification on such detected clickbait posts,
we conduct three experiments: (1) multi-class, (2) one-vs-rest, and (3) one-
vs-one for the types of phrase and passage spoilers.

Table 2: Effectiveness of spoiler type classification in the multi-class (first column) and one-vs-rest settings on 1000 test posts (training: 3200; validation: 800). Model |  Balanced accuracy (0, 1, 2 indicate class labels)
---|---
| Phrase | 0 | 1 | 0 | 0
| Passage | 1 | 0 | 1 | 0
| Multipart | 2 | 0 | 0 | 1
Naïve Bayes | 56.15 | 65.03 | 62.50 | 64.82
SVM | 59.62 | 68.03 | 68.70 | 70.28
Log. Regression | 60.04 | 68.04 | 69.33 | 71.26
BERT | 67.84 | 74.06 | 75.70 | 75.56
DeBERTa | 73.63 | 78.39 | 78.65 | 77.93
RoBERTa | 71.57 | 80.39 | 79.30 | 79.12

In all cases, the hyperparameters of the six studied classifiers were
optimized based on the validation set of our corpus. For the three feature-
based approaches, a chi-square feature selection step selected all post-based
features and 70% of the document-based features. The post-based features are
weighted 4-times higher than the document-based features. Most hyperparameters
of the transformer models were left at their default values, but a grid search
was used to find the most effective combination of learning rate (1e-5, 4e-5,
1e-4), warm-up ratio (0.02, 0.06, and 0.1), stack size (8, 16, and 32), number
of epochs (1 to 10), and maximum sequence length (256, 384, 512).

Table 2 shows the balanced accuracy of the six classifiers. All are less
effective in the multi-class setting than in the one-vs-rest settings and the
transformer-based classifiers are clearly more effective than the feature-
based ones; DeBERTa is best in the multi-class setting (accuracy of 73.63) and
RoBERTa in the one-vs-rest ones (79.12 to 80.39).

Table 3: Effectiveness of spoiler type classification in the one-vs-one (phrase-vs-passage) setting on 826 test posts (training: 2,641; validation: 657). Model | Effectiveness
---|---
| TP | TN | FP | FN | Acc.
Naïve Bayes | 298 | 256 | 147 | 125 | 67.07
SVM | 311 | 264 | 139 | 112 | 69.61
Log. Regression | 306 | 273 | 130 | 117 | 70.10
BERT | 315 | 315 |  088 | 108 | 76.27
DeBERTa | 318 | 335 |  068 | 105 | 79.06
RoBERTa | 332 | 332 |  071 |  091 | 80.39

Table 3 shows the accuracy of the six classifiers on the 826 test posts with
phrase and passage spoilers (almost balanced setup, since there is hardly any
class imbalance). Again, the transformer-based classifiers clearly are more
effective than the feature-based ones; with RoBERTa achieving the best
accuracy of 80.39.

The substantial improvements of DeBERTa and RoBERTa over the feature-based
classifiers in all settings (about 9–10 accuracy points) indicates that
classifying the clickbait spoiler type requires more advanced language
“understanding” than what is encoded in the basic features that the Naïve
Bayes, SVM, or logistic regression classifiers used.

##  6 Evaluation of Spoiler Generation

To assess the effectiveness of the question answering and passage retrieval
methods for clickbait spoiling, we evaluate both for their respective intended
spoiler types, but each also for the respective other spoiler type. Multipart
spoilers are deferred to future work. We continue to assume that prior
clickbait detection (perfectly) identifies clickbait posts as such. Our
evaluation of the generated spoilers includes quantitative and qualitative
assessments (Section 6.1). In a pilot study with ten question answering and
ten passage retrieval models at their default settings, two models in each
category dominate the respective others (Section 6.2). The computationally
expensive step of hyperparameter optimization is restricted to these four
models plus two baselines (Section 6.3). Then, the effectiveness of spoiling
clickbait posts dependent on spoiler type is evaluated (Sections 6.4 and 6.5),
and compared to an end-to-end clickbait spoiling setup independent of spoiler
type (section 6.6).

###  6.1 Quantitative and Qualitative Assessment

We introduce the measures used to evaluate generated spoilers and how we
manually determined thresholds for them above which a generated spoiler is
considered as “correct”.

#### Evaluation measures.

To assess the quantitative correspondence between a derived spoiler and the
ground truth, we use three question answering-oriented and one passage
retrieval-oriented measure: BLEU-4 Papineni et al. (2002), METEOR Banerjee and
Lavie (2005) in its extended version of Denkowski and Lavie (2014), BERTScore
Zhang et al. (2020), and Precision@1.

The three question answering-oriented measures each calculate a (penalized)
harmonic mean of measure-specific definitions of precision and recall when
comparing a generated spoiler to the ground truth. In case of BLEU-4, the
overlap of word 1- to 4-grams is determined (if the length $n$ of a generated
spoiler is less than 4 words, we compute BLEU-$n$), in case of METEOR the
overlap of word 1-grams, and in case of BERTScore the best matching embeddings
of word pairs. Note that in their original formulation, BLEU-4 and METEOR
penalize the score, the more the $n$-gram order differs. To arrange the
measures on a spectrum from calculating predominantly syntactic (BLEU-4) to
predominantly semantic similarity (BERTScore), we omit METEOR’s penalization
term.

The question answering-oriented measures are not really suited to assess the
effectiveness of passage retrieval models since a retrieved passage is often
longer than the ground truth spoiler. Therefore, we also use Precision@1 to
measure whether the top-ranked passage contains the ground truth spoiler (all
phrase spoilers and 98% of the passage spoilers come from a single passage;
for the other passage spoilers, we consider all containing passages as
relevant). To calculate the Precision@1 of question answering models, we use
the first passage that contains the returned spoiler.

Table 4: Manually determined numbers of false positives/negatives (FP/FN) on 500 sampled clickbait posts with phrase spoilers and 500 with passage spoilers for question answering (top row group) and passage retrieval models (bottom row group), dependent on score threshold (Thresh.), spoiler type, and effectiveness measure (BL4 = BLEU-4, MET = METEOR, BSc. = BERTScore). The thresholds selected for subsequent assessment are indicated by bold FP/FN numbers. Thresh. | Phrase Spoilers | Passage Spoilers
---|---|---
| BL4 | MET | BSc. | BL4 | MET | BSc.
| FP | FN | FP | FN | FP | FN | FP | FN | FP | FN | FP | FN
10% | 11 | 11 | 18 | 7 | 238 | 0 | 5 | 44 | 168 | 15 | 399 | 0
20% | 7 | 14 | 16 | 7 | 234 | 0 | 3 | 48 | 67 | 27 | 325 | 3
30% | 7 | 14 | 14 | 9 | 165 | 1 | 1 | 51 | 31 | 35 | 134 | 21
40% | 2 | 27 | 8 | 13 | 59 | 6 | 0 | 55 | 15 | 39 | 18 | 38
50% | 2 | 27 | 2 | 28 | 24 | 14 | 0 | 60 | 9 | 42 | 5 | 51
60% | 2 | 30 | 3 | 31 | 11 | 25 | 0 | 64 | 4 | 57 | 1 | 59
70% | 1 | 33 | 2 | 31 | 6 | 36 | 0 | 66 | 1 | 54 | 0 | 66
80% | 1 | 34 | 0 | 37 | 1 | 40 | 0 | 66 | 0 | 61 | 0 | 73
5% | 8 | 40 | 28 | 64 | 208 | 0 | 0 | 95 | 225 | 10 | 355 | 0
10% | 4 | 104 | 8 | 108 | 180 | 60 | 0 | 95 | 140 | 30 | 355 | 0
20% | 0 | 184 | 0 | 164 | 44 | 144 | 0 | 95 | 35 | 65 | 305 | 15
30% | 0 | 188 | 0 | 184 | 0 | 176 | 0 | 105 | 5 | 90 | 145 | 55
40% | 0 | 188 | 0 | 188 | 0 | 188 | 0 | 115 | 5 | 105 | 20 | 95
50% | 0 | 192 | 0 | 188 | 0 | 192 | 0 | 120 | 5 | 110 | 5 | 105
60% | 0 | 192 | 0 | 192 | 0 | 192 | 0 | 125 | 0 | 120 | 5 | 130
Table 5: Pilot study spoiling effectiveness of question answering and passage retrieval models on 200 validation posts (models ordered lexicographically). The bracketed numbers indicate the expected number of true positives as per our pre-determined high-confidence score thresholds; P@1 is the Precision@1. The models DeBERTa-large and RoBERTa-large, as well as MonoBERT and MonoT5 are the most effective in their groups. Type | Model |  Phrase Spoilers ($n$ = 97) |  Passage Spoilers ($n$ = 103)
---|---|---|---
|  | BLEU-4 | METEOR | BERTScore | P@1 | BLEU-4 | METEOR | BERTScore | P@1
Question Answering | ALBERT | 63.82 (50) | 55.97 (49) | 74.07 (46) | 63.64 | 24.51 (33) | 38.42 (27) | 44.61 (24) | 38.71
BERT-cased | 60.27 (49) | 58.87 (47) | 73.55 (44) | 59.09 | 17.65 (22) | 28.09 (20) | 40.30 (16) | 27.96
BERT-uncased | 62.36 (49) | 53.17 (47) | 75.87 (47) | 60.23 | 18.05 (22) | 32.50 (20) | 39.86 (18) | 32.26
Big Bird | 69.21 (55) | 64.80 (54) | 77.39 (49) | 63.64 | 23.89 (30) | 36.20 (28) | 44.55 (27) | 43.01
DeBERTa-large | 70.19 (57) | 65.08 (56) | 78.02 (50) | 65.91 | 29.52 (38) | 43.72 (36) | 49.63 (37) | 48.39
ELECTRA | 69.10 (55) | 65.97 (53) | 79.26 (51) | 65.91 | 25.78 (32) | 39.87 (30) | 46.64 (27) | 43.01
Funnel-Transf. | 68.31 (54) | 63.89 (53) | 78.78 (51) | 64.77 | 28.59 (36) | 40.95 (32) | 47.93 (29) | 40.86
MPNet | 72.92 (58) | 65.90 (57) | 80.26 (55) | 69.32 | 30.16 (36) | 40.68 (35) | 50.07 (32) | 40.86
RoBERTa-base | 73.02 (59) | 65.56 (57) | 80.39 (54) | 65.91 | 27.61 (35) | 41.55 (35) | 48.76 (30) | 44.09
RoBERTa-large | 79.47 (66) | 78.61 (61) | 84.04 (58) | 70.45 | 29.58 (35) | 43.49 (32) | 48.65 (32) | 44.09
Passage Retrieval | BM25 |  03.49 (10) |  03.67 (10) | 17.73 0(2) | 5.68 | 11.49 (22) | 22.64 (21) | 36.80 (12) | 9.68
BM25+Ax |  03.39 (10) |  03.57 0(9) | 18.07 0(2) | 5.68 | 11.27 (21) | 22.46 (19) | 36.51 (12) | 9.94
BM25+PRF |  03.25 (10) |  03.21 0(9) | 18.03 0(2) | 5.13 |  09.68 (20) | 21.10 (17) | 35.44 (11) | 8.84
BM25+RM3 |  03.43 (10) |  03.62 0(9) | 17.14 0(2) | 5.13 | 10.06 (21) | 21.03 (20) | 35.56 (11) | 8.84
MonoBERT |  03.42 (11) |  04.13 (12) | 18.32 0(1) | 32.95 | 14.55 (29) | 26.86 (25) | 38.10 (15) | 31.18
MonoT5 |  03.16 0(9) |  04.19 (11) | 18.30 0(0) | 31.82 | 14.27 (29) | 26.70 (26) | 38.94 (17) | 29.03
QLD |  02.51 0(7) |  02.69 0(7) | 17.24 0(0) | 12.50 | 10.94 (25) | 17.80 (18) | 36.70 (11) | 19.35
QLD+Ax |  02.61 0(7) |  02.71 0(7) | 17.10 0(0) | 12.50 |  09.68 (20) | 17.84 (18) | 36.68 (11) | 8.84
QLD+PRF |  02.60 0(7) |  02.70 0(7) | 17.13 0(0) | 11.94 | 10.86 (25) | 17.52 (18) | 36.46 (11) | 17.67
QLD+RM3 |  02.41 0(7) |  02.54 0(7) | 16.97 0(0) | 11.39 | 10.66 (25) | 17.54 (18) | 36.13 (11) | 17.12

#### High-confidence thresholds.

Candidates with higher scores on the question answering-oriented measures
BLEU-4, METEOR, and BERTScore are closer to the ground truth. However, it is
unclear what score threshold a particular spoiler candidate has to exceed so
that it would be considered a true positive in a manual analysis. Determining
such thresholds enables “high confidence” estimations of how many correct
spoilers an approach generates without having to manually check its outputs
each time with each new variant.

In a pilot study, we thus determined such thresholds by running all question
answering models (cf. Section 4.2 and 4.3) on a random sample of 500 clickbait
posts with phrase spoilers and 500 with passage spoilers. For each post, a
random spoiler generated by a question answering model and a random spoiler
generated by a passage retrieval model were manually checked for whether they
could be viewed as correct. Table 4 shows the number of manually determined
false positives and false negatives for different thresholds of BLEU-4,
METEOR, and BERTScore. The manually selected subjective thresholds (FP/FN in
bold) for each combination of measure, spoiler type, and model type (question
answering or passage retrieval) minimize the false positives at a rate where
being more strict would incur too many false negatives. For instance, for
phrase spoilers and BLEU-4, we set the question answering model threshold at
50% since a more strict threshold of 60% does not reduce the false positives
but increases the false negatives.

In addition to reporting quantitative mean effectiveness scores, applying the
determined thresholds helps to estimate how many of the spoilers of a model
would be perceived as “good” by human readers. This corresponds to a
conservative assessment, since we believe that a model should only be deployed
to production if it has been tuned to not return a spoiler if in doubt about
its correctness; also probably somewhat minimizing the otherwise possible
spread of auto-generated misinformation.

###  6.2 Pilot Study for Model Selection

In a pilot study on 1,000 clickbait posts (800 training, 200 validation), we
compare ten question answering and ten passage retrieval models (cf. Table 5)
at their default settings to select models for subsequent experiments with
more extensive (and expensive) hyperparameter tuning. The question answering
models were or are among the most effective in the SQuAD and TriviaQA question
answering benchmarks. In our setup, they return a piece of text from the
linked document as an “answer” to the clickbait post as the “query”. As
passage retrieval models, we empoly MonoBERT and MonoT5 using their
PyGaggle333https://github.com/castorini/pygaggle implementations, and eight
variants of the popular baseline retrieval models BM25 and QLD using their
Anserini implementations Yang et al. (2017). These models return the most
“relevant” paragraph from the linked document for the clickbait post as the
“query”.

Table 6: Effectiveness on the 826 test clickbait posts with phrase and passage spoilers. The bracketed numbers indicate the expected number of true positives as per our pre-determined high-confidence score thresholds; P@1 is the Precision@1. Overall, DeBERTa-large and RoBERTa-large are the most effective models. Type | Model |  Phrase Spoilers ($n$ = 423) |  Passage Spoilers ($n$ = 403)
---|---|---|---
|  | BLEU-4 | METEOR | BERTScore | P@1 | BLEU-4 | METEOR | BERTScore | P@1
Question Answering | BERT (baseline) | 58.89 (257) | 56.75 (266) | 71.06 (215) | 66.67 | 21.59 (110) | 35.49 (100) | 44.38 (109) | 42.43
DeBERTa-large | 68.80 (300) | 67.93 (298) | 77.03 (250) | 75.65 | 31.44 (157) |  46.06 (142) |  51.06 (161) | 54.84
RoBERTa-large | 65.70 (290) | 66.15 (293) | 74.81 (233) | 72.58 | 29.61 (148) | 45.20 (145) | 49.99 (167) | 53.85
Passage Retrieval | BM25 (baseline) |  03.40 0(55) |  05.06 0(83) | 19.94 0(12) | 8.27 |  07.91 0(53) | 20.19 0(61) | 34.71 0(42) | 4.22
MonoBERT |  04.20 0(72) |  06.12 (103) | 20.66 0(11) | 42.08 | 10.43 0(74) | 22.37 0(75) | 36.58 0(46) | 26.05
MonoT5 |  04.95 0(82) |  06.47 (115) | 20.98 0(16) | 43.97 | 10.58 0(74) | 22.02 0(74) | 36.70 0(46) | 29.03

Using Nvidia A100 GPUs, the question answering models were first fine-tuned on
SQuAD v1.1 and then on the pilot training data. This was the most effective
setup from an ablation study with other fine-tuning regimes (e.g., the phrase
spoiler BERTScore for RoBERTa-large dropped from 84.04 to 69.91 when only
fine-tuned on our pilot study data, to 64.61 when only fine-tuned on SQuAD,
and to 46.60 without fine-tuning). Interestingly, the models’ SQuAD
effectiveness does not predict their spoiling effectiveness (e.g., RoBERTa-
base and FunnelTransformer were tied on SQuAD, but RoBERTa-base is more
effective at spoiling). This indicates the importance of the pilot study.

Table 5 shows the pilot study effectiveness of all models on the 200
validation posts. RoBERTa-large (for phrasal spoilers) and DeBERTa-large (for
passage spoilers) are the most effective. Among the passage retrieval models,
MonoBERT and MonoT5 achieve the best scores. Contrary to our original
assumption that passage retrieval models might be particularly well-suited to
identify passage spoilers, MonoBERT and MonoT5 have similar Precision@1 scores
on both phrase and passage spoilers and are substantially less effective than
the best question answering models (e.g., DeBERTa-large has a Precision@1 of
48.39 for passage spoilers compared to 31.18 for MonoBERT).

###  6.3 Tuning the Selected Models

Given the pilot study results, six models are selected for a more extensive
hyperparameter tuning: the best two question answering models (DeBERTa-large
was best for phrase spoilers, RoBERTa-large for passage spoilers) plus BERT as
baseline, as well as the best two passage retrieval models (MonoBERT and
MonoT5) plus BM25 as baseline.

As the ablation study in our pilot study showed that fine-tuning the question
answering models on SQuAD first and then on our corpus works best, we apply
this fine-tuning regime to DeBERTa-large, RoBERTa-large, and BERT using the
clickbait spoiling training data (depending on the experiment, either only the
phrase spoilers, only the passage spoilers, or both combined). Most
hyperparameters of DeBERTa-large, RoBERTa-large, BERT, MonoBERT, and MonoT5
are left at their defaults, but a grid search is run to find the most
effective combination of learning rate (1e-5, 4e-5, 1e-4), warmup ratio (0.02,
0.06, 0.1), batch size (8, 16, 32), number of epochs (1 to 10), and maximum
sequence length (256, 384, 512). For BM25, we try combinations of $k_{1}$ from
0.1 to 0.4 and $b$ from 0.1 to 1.0 with a step size of 0.1.

###  6.4 Effectiveness on Phrase Spoilers

The ‘Phrase Spoilers’ column group in Table 6 shows the effectiveness of the
selected question answering and passage retrieval models on the 423 test
clickbait posts with phrase spoilers. Given the ground-truth spoiler, we
report the predicted spoilers’ average BLEU-4, METEOR, BERTScore, and
Precision@1 (using 1,367 posts with phrase spoilers for training and 335 posts
for validation to tune the hyperparameters; cf. Table 1).

Overall, DeBERTa-large is the most effective model for phrase spoilers. Based
on our high-confidence score thresholds, it generates the correct spoiler for
250–300 of the 423 test posts (i.e., for about 60–70% of the cases) according
to a BERTScore or BLEU-4 evaluation. Similar to our pilot study, the passage
retrieval models are comparably ineffective in identifying phrase spoilers.
Among them, MonoT5 achieves the highest scores but is even substantially less
effective than the question answering baseline BERT. For instance, with a
BLEU-4 of 58.89 and probably 257 correct spoilers (61% of the 423 test posts),
BERT is way ahead of MonoT5 with a BLEU-4 of 4.95 and only 82 probably correct
spoilers (19% of the 423 posts).

###  6.5 Effectiveness on Passage Spoilers

The ‘Passage Spoilers’ column group in Table 6 shows the effectiveness of the
selected passage retrieval models on the 403 test clickbait posts with passage
spoilers (using 1,274 and 322 posts for training and validation). The numbers
of probably correct spoilers are lower for all models compared to the phrase
spoilers (even the higher amount of probably correct passage spoilers of the
passage retrieval models according to their BERTScore threshold are still
worse than the estimated probably correct phrase spoilers according to BLEU-4
or METEOR). Similar to the pilot study, all question answering models are also
substantially more effective on passage spoilers than the passage retrieval
models. Overall, DeBERTa-large and RoBERTa-large achieve the highest
Precision@1 scores and the highest amount of probably correct passage spoilers
(about 35–41% of the passage spoilers are correctly identified according to
our high-confidence thresholds).

###  6.6 Effectiveness of the End-to-End System

Table 7: End-to-end effectiveness on the 826 phrase and passage test posts. Spoiling models that classify the spoiler type to then select an appropriately trained spoiler model (‘Classif.’, using the most effective spoiler type classifier), models without spoiler type classification (‘None’), and unrealistic models with perfect-accuracy type classification (‘Oracle’). | Model | End-to-End Effectiveness
---|---|---
|  | BLEU-4 | METEOR | BERTScore | P@1
Classif. | BERT | 35.95 (311) | 34.25 (303) | 53.86 (294) | 52.66
DeBERTa | 44.98 (392) | 44.32 (377) | 59.18 (378) | 63.44
RoBERTa | 42.70 (374) | 43.23 (356) | 58.01 (361) | 61.86
None | BERT | 38.85 (346) | 37.80 (330) | 54.60 (314) | 55.33
DeBERTa | 46.16 (409) | 47.01 (407) | 60.43 (382) | 64.16
RoBERTa | 44.69 (400) | 44.72 (395) | 59.51 (375) | 65.13
Oracle | BERT | 40.69 (367) | 39.02 (366) | 58.05 (324) | 54.84
DeBERTa | 50.58 (457) | 49.40 (440) | 64.36 (411) | 65.50
RoBERTa | 48.10 (438) | 48.57 (438) | 62.71 (400) | 63.44

We evaluate the entire spoiling pipeline using all 826 phrase and passage test
posts by comparing two-step pipelines that first classify the spoiler type to
then select an appropriately trained spoiler model (trained on the respective
type) and single-step approaches that skip the spoiler type classification and
simply run the same spoiler model on all posts (trained on the complete
training data). For the two-step pipelines, we experiment with two variants:
(1) using an artificial classifier that returns perfect oracle-style answers
about a post’s type, and (2) using the best RoBERTa-based phrase-vs-passage
classifier from Section 5.

Since the passage retrieval models were less effective in our spoiler
experiments (cf. Table 6), we report results only for pipelines with question
answering models. In the two-step pipelines the respective question answering
models are fine-tuned on the respective spoiler types, in the single-step
approach on the combined training data.

Table 7 shows the achieved end-to-end effectiveness values. The individual
two-step pipelines with oracle type classification (row group ‘Oracle’) are
substantially more effective than their single-step counterparts without type
classification (row group ‘None’) that again are more effective than the
respective two-step pipelines with “real” RoBERTa-based type classification
(row group ‘Classif.’). Overall, the DeBERTa pipeline with oracle classifier
achieves an estimated amount of about 50–55% correctly spoiled posts (i.e.,
411 to 457 of 826). This result confirms that classifying the required spoiler
type can be beneficial for clickbait spoiling. Still, among the currently
realistically applicable end-to-end spoiling approaches (with RoBERTa type
classification or without spoiler type classification), the one-step DeBERTa
approach without spoiler type classification is the most effective according
to the number of probably correctly spoiled posts (382 to 409 of the 826
posts, i.e., 46–50%). This indicates that the currently best RoBERTa-based
spoiler type classifier with its accuracy of 80.39% is still not good enough
to result in an end-to-end system that actually benefits from spoiler type
classification.

Our results show that effectively spoiling clickbait with question answering
models is possible in practice but also that there is still room for
improvements (e.g., improved spoiler type classification, improved spoiler
generation for the individual types, and taking multipart spoilers into
account).

##  7 Conclusion

Clickbait spoiling is a new task to help social media users who do not want to
be manipulated into falling for clickbait links. Unlike clickbait detection,
which often involves filtering out clickbait posts from users’ timelines,
clickbait spoiling subverts the curiosity triggered by clickbait, presenting
users with the withheld “punchline” in advance.

We compile the first large resource for clickbait with associated spoilers. By
interpreting clickbait spoiling as either a question answering task or a
passage retrieval task, many possible approaches are available to extract from
the linked document of a clickbait post the phrase or passage that spoils it.
We have explored the effectiveness of a number of state-of-the-art solutions
for both tasks in a large-scale experiment, including fine-tuning the
respective models on our resource to determine their effectiveness for type-
specific clickbait spoiling. Our experimental setup considers type-specific
spoiling on the one hand, but on the other hand it also includes an end-to-end
configuration for comparison. Overall, our results show that type-agnostic
question answering-based spoiling is the most effective yet, but also that
spoiler type-specific solutions have the potential to outperform them.

In addition to the possibilities explored, there might also be other
approaches to clickbait spoiling: for example, paraphrasing technology could
be used to directly transform a clickbait post into a version that contains
its own spoiler. With respect to multipart spoilers, the use of summarization
models could be an interesting direction to select the different parts of the
linked document of a clickbait post that make up its multipart spoiler.

## Acknowledgement

We thank Tim Gollub, and our students Jana Puschmann and Bagrat Ter-Akopyan,
who helped to create earlier versions of the dataset.

## Ethics Statement

The spread of clickbait on social media by news publishers to promote click-
through to their websites has been empirically found to decrease their
perceived credibility in readers Molyneux and Coddington (2020). There is, of
course, nothing wrong with monitoring and optimizing the effectiveness of
marketing a newly published news article, especially in cases where the
editors make an honest effort to reach and inform their target audience. But
the clickbait in our corpus mostly spreads trivial facts that could have been
easily fitted into the length limits of a social media post, which is why we
consider these posts to fall short of the journalistic ideal. However, it is
as of yet unclear, in terms of journalism ethics, whether clickbait is an
acceptable means to an end for publishers (i.e., whether it is “necessary in
driving audiences to the journalism they need by giving them the journalism
they seem to want.”), or whether it serves to “crowding out «real» journalism
by reducing quality in favor of the need for a click-through at whatever cost”
Harte (2021).

Facebook intervened twice with algorithmic filters to reduce the amount of
clickbait that people are exposed to in their timelines—even though this
probably also lowered Facebook’s user engagement metrics. Our technology
demonstrates another, complementary way of relatively simply circumventing the
purported exploitation of the curiosity gap by giving the audience a choice on
whether or not they wish their cognitive “loopholes” to be exploited. If a
sufficiently large portion of people decide to adopt spoiling tools, that
would send a clear message to publishers and social media platforms alike.
Spoiling clickbait, as opposed to removing it, however, still gives publishers
the benefit of the doubt, since, as the publishers claim, there are people who
enjoy these kinds of trivia.

## References

  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments.  In _Proceedings of the Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization@ACL 2005, Ann Arbor, Michigan, USA, June 29, 2005_ , pages 65–72. Association for Computational Linguistics.
  * Chakraborty et al. (2016) Abhijnan Chakraborty, Bhargavi Paranjape, Sourya Kakarla, and Niloy Ganguly. 2016\.  Stop Clickbait: Detecting and Preventing Clickbaits in Online News Media.  In _2016 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2016, San Francisco, CA, USA, August 18-21, 2016_ , pages 9–16.
  * Clark and Gardner (2018) Christopher Clark and Matt Gardner. 2018.  Simple and Effective Multi-Paragraph Reading Comprehension.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers_ , pages 845–855. Association for Computational Linguistics.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020.  ELECTRA: pre-training text encoders as discriminators rather than generators.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Craswell et al. (2020) Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. 2020.  Overview of the TREC 2020 Deep Learning Track.  In _Proceedings of the Twenty-Ninth Text REtrieval Conference, TREC 2020, Virtual Event [Gaithersburg, Maryland, USA], November 16-20, 2020_ , volume 1266 of _NIST Special Publication_. National Institute of Standards and Technology (NIST).
  * Craswell et al. (2019) Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M. Voorhees. 2019.  Overview of the TREC 2019 Deep Learning Track.  In _Proceedings of the Twenty-Eighth Text REtrieval Conference, TREC 2019, Gaithersburg, Maryland, USA, November 13-15, 2019_ , volume 1250 of _NIST Special Publication_. National Institute of Standards and Technology (NIST).
  * Dai et al. (2020) Zihang Dai, Guokun Lai, Yiming Yang, and Quoc Le. 2020.  Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Denkowski and Lavie (2014) Michael J. Denkowski and Alon Lavie. 2014.  Meteor Universal: Language Specific Translation Evaluation for Any Target Language.  In _Proceedings of the Ninth Workshop on Statistical Machine Translation, WMT@ACL 2014, June 26-27, 2014, Baltimore, Maryland, USA_ , pages 376–380. The Association for Computer Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 4171–4186. Association for Computational Linguistics.
  * Dzendzik et al. (2021) Daria Dzendzik, Jennifer Foster, and Carl Vogel. 2021.  English Machine Reading Comprehension Datasets: A Survey.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_ , pages 8784–8804. Association for Computational Linguistics.
  * Eidnes (2015) Lars Eidnes. 2015.  Auto-Generating Clickbait With Recurrent Neural Networks.  https://web.archive.org
/web/20220223161935/http://larseidnes.com/2015/
10/13/auto-generating-clickbait-with-recurrent-
neural-networks/.

  * El-Arini and Tang (2014) Khalid El-Arini and Joyce Tang. 2014.  News Feed FYI: Click-baiting.  http://web.archive.org/web/2015052
9104738/http://newsroom.fb.com/news/2014/08/
news-feed-fyi-click-baiting/.

  * Gokaslan and Cohen (2019) Aaron Gokaslan and Vanya Cohen. 2019.  OpenWebText Corpus.  http://Skylion007.github.io/OpenWeb
TextCorpus.

  * Guo et al. (2020) Jiafeng Guo, Yixing Fan, Liang Pang, Liu Yang, Qingyao Ai, Hamed Zamani, Chen Wu, W. Bruce Croft, and Xueqi Cheng. 2020.  A Deep Look Into Neural Ranking Models for Information Retrieval.  _Inf. Process. Manag._ , 57(6):102067.
  * Harte (2021) David Harte. 2021.  Clickbait and Banal News.  In _The Routledge Companion to Journalism Ethics_ , pages 346–353. Routledge.
  * He et al. (2021) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021.  DeBERTa: Decoding-Enhanced BERT with Disentangled Attention.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017.  TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers_ , pages 1601–1611. Association for Computational Linguistics.
  * Karn et al. (2019) Sanjeev Kumar Karn, Mark Buckley, Ulli Waltinger, and Hinrich Schütze. 2019\.  News Article Teaser Tweets and How to Generate Them.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 3967–3977. Association for Computational Linguistics.
  * Koechley (2012) Peter Koechley. 2012.  Why The Title Matters More Than The Talk.  http://web.archive.org/web/2015061
1110506/http://blog.upworthy.com/post/263456340
89/why-the-title-matters-more-than-the-talk.

  * Lan et al. (2020) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020.  ALBERT: A Lite BERT for Self-supervised Learning of Language Representations.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Lin et al. (2021) Jimmy Lin, Rodrigo Nogueira, and Andrew Yates. 2021.  _Pretrained Transformers for Text Ranking: BERT and Beyond_.  Synthesis Lectures on Human Language Technologies. Morgan & Claypool Publishers.
  * Lischka and Garz (2021) Juliane A Lischka and Marcel Garz. 2021.  Clickbait News and Algorithmic Curation: A Game Theory Framework of the Relation between Journalism, Users, and Platforms.  _New Media & Society_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  _CoRR_ , abs/1907.11692.
  * Loewenstein (1994) George Loewenstein. 1994.  The Psychology of Curiosity: A Review and Reinterpretation.  _Psychological Bulletin_ , 116(1):75–98.
  * Molyneux and Coddington (2020) Logan Molyneux and Mark Coddington. 2020.  Aggregation, Clickbait and Their Effect on Perceptions of Journalistic Credibility and Quality.  _Journalism Practice_ , 14(4):429–446.
  * Munger (2020) Kevin Munger. 2020.  All the News That’s Fit to Click: The Economics of Clickbait Media.  _Political Communication_ , 37(3):376–397.
  * Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016.  MS MARCO: A Human Generated MAchine Reading COmprehension Dataset.  In _Proceedings of the Workshop on Cognitive Computation: Integrating neural and symbolic approaches 2016 co-located with the 30th Annual Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain, December 9, 2016_ , volume 1773 of _CEUR Workshop Proceedings_. CEUR-WS.org.
  * Nogueira and Cho (2019) Rodrigo Nogueira and Kyunghyun Cho. 2019.  Passage Re-ranking with BERT.  _CoRR_ , abs/1901.04085.
  * Nogueira et al. (2020) Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. 2020.  Document Ranking with a Pretrained Sequence-to-Sequence Model.  In _Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020_ , volume EMNLP 2020 of _Findings of ACL_ , pages 708–718. Association for Computational Linguistics.
  * Nogueira et al. (2019) Rodrigo Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy Lin. 2019.  Multi-Stage Document Ranking with BERT.  _CoRR_ , abs/1910.14424.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  BLEU: A Method for Automatic Evaluation of Machine Translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA_ , pages 311–318. ACL.
  * Peysakhovich and Hendrix (2016) Alex Peysakhovich and Kristin Hendrix. 2016.  Further Reducing Clickbait in Feed.  https://web.archive.org/
web/20210207042429/https://about.fb.com/news/20
16/08/news-feed-fyi-further-reducing-clickbait-in-
feed/.

  * Ponte and Croft (1998) Jay M. Ponte and W. Bruce Croft. 1998.  A Language Modeling Approach to Information Retrieval.  In _SIGIR ’98: Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, August 24-28 1998, Melbourne, Australia_ , pages 275–281. ACM.
  * Potthast et al. (2018) Martin Potthast, Tim Gollub, Matthias Hagen, and Benno Stein. 2018.  The Clickbait Challenge 2017: Towards a Regression Model for Clickbait Strength.  _CoRR_ , abs/1812.10847.
  * Potthast et al. (2016) Martin Potthast, Sebastian Köpsel, Benno Stein, and Matthias Hagen. 2016.  Clickbait Detection.  In _Advances in Information Retrieval. 38th European Conference on IR Research (ECIR 2016)_ , volume 9626 of _Lecture Notes in Computer Science_ , pages 810–817, Berlin Heidelberg New York. Springer.
  * Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018.  Know What You Don’t Know: Unanswerable Questions for SQuAD.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, 2: Short Papers_ , pages 784–789. Association for Computational Linguistics.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ Questions for Machine Comprehension of Text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016_ , pages 2383–2392. The Association for Computational Linguistics.
  * Robertson and Zaragoza (2009) Stephen E. Robertson and Hugo Zaragoza. 2009.  The Probabilistic Relevance Framework: BM25 and Beyond.  _Found. Trends Inf. Retr._ , 3(4):333–389.
  * Rony et al. (2017) Md M. U. Rony, Naeemul Hassan, and Mohammad Yousuf. 2017.  Diving Deep into Clickbaits: Who Use Them to What Extents in Which Topics with What Effects? In _Proceedings of the 2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining 2017, Sydney, Australia, July 31 - August 03, 2017_ , pages 232–239. ACM.
  * Rubin et al. (2015) Victoria Rubin, Niall Conroy, and Yimin Chen. 2015.  Towards News Verification: Deception Detection Methods for News Discourse.  In _Proceedings of the Hawaii International Conference on System Sciences (HICSS48) Symposium on Rapid Screening Technologies, Deception Detection and Credibility Assessment Symposium_ , Kauai, Hawaii, USA.
  * Shu et al. (2018) Kai Shu, Suhang Wang, Thai Le, Dongwon Lee, and Huan Liu. 2018.  Deep Headline Generation for Clickbait Detection.  In _IEEE International Conference on Data Mining, ICDM 2018, Singapore, November 17-20, 2018_ , pages 467–476. IEEE Computer Society.
  * Song et al. (2020) Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020.  MPNet: Masked and Permuted Pre-training for Language Understanding.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Xu et al. (2019) Peng Xu, Chien-Sheng Wu, Andrea Madotto, and Pascale Fung. 2019.  Clickbait? Sensational Headline Generation with Auto-tuned Reinforcement Learning.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , pages 3063–3073. Association for Computational Linguistics.
  * Yang et al. (2017) Peilin Yang, Hui Fang, and Jimmy Lin. 2017.  Anserini: Enabling the Use of Lucene for Information Retrieval Research.  In _Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, Shinjuku, Tokyo, Japan, August 7-11, 2017_ , pages 1253–1256. ACM.
  * Zaheer et al. (2020) Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020.  Big Bird: Transformers for Longer Sequences.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020\.  BERTScore: Evaluating Text Generation with BERT.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.