# Orchestrating time and color: a programmable source of high-dimensional
entanglement

Laura Serino  1,${\dagger}$,* Werner Ridder  1,${\dagger}$ Abhinandan
Bhattacharjee  1 Jano Gil-Lopez  1 Benjamin Brecht  1 and Christine
Silberhorn1  1Paderborn University, Integrated Quantum Optics, Institute for
Photonic Quantum Systems (PhoQS), Warburgerstr. 100, 33098 Paderborn, Germany
${\dagger}$The authors contributed equally to this work<EMAIL_ADDRESS>
††journal: opticajournal††articletype: Research Article

High-dimensional encodings based on temporal modes (TMs) of photonic quantum
states provide the foundations for a highly versatile and efficient quantum
information science (QIS) framework. Here, we demonstrate a crucial building
block for any QIS applications based on TMs: a programmable source of
maximally entangled high-dimensional TM states. Our source is based on a
parametric down-conversion process driven by a spectrally shaped pump pulse,
which facilitates the generation of maximally entangled TM states with a well-
defined dimensionality that can be chosen programmatically. We characterize
the effective dimensionality of the generated states via measurements of
second-order correlation functions and joint spectral intensities,
demonstrating the generation of bi-photon TM states with a controlled
dimensionality in up to 20 dimensions.

##  1 Introduction

In recent years, the field of quantum communication has experienced remarkable
progress, bringing us closer to the creation of a so-called quantum internet
[1] which will exploit the fundamental properties of quantum particles to
guarantee secure and efficient transmission of information. High-dimensional
entangled states play a key role in these advancements [2]: their higher
information capacity allows for significantly more efficient communication,
and high-dimensional quantum cryptography protocols offer enhanced security
[3]. Photons emerge as a natural information carrier due to their inherent
quantum nature and high-dimensional spatial and time-frequency degrees of
freedom.

Of the high-dimensional photonic degrees of freedom, the spatial domain is
arguably the most explored, due to the possibility to generate and manipulate
states using only time-invariant operations [4, 5, 6]. However, spatial
encoding is incompatible with existing single-mode fiber networks and
information are easily degraded by turbulence in free-space transmission [7].

Encoding information in the time-frequency domain of photons overcomes these
disadvantages by offering robust transmission both through optical fiber and
free space. In this platform, information can be encoded in pulsed temporal
modes (TMs) [8], i.e., field-orthogonal wave-packet modes. A practical quantum
communication framework based on TMs requires tools to generate, manipulate
and detect high-dimensional TM states. The manipulation and detection of one
or many single-photon TMs have been demonstrated through the so-called quantum
pulse gate [9, 10, 11], interferometric systems [12] or a combination of phase
modulators and pulse shapers [13, 14]. However, the realization of an optimal
source of high-dimensional entangled TM states remains an open challenge.

An ideal source of time-frequency entanglement must generate bi-photon states
with a well-defined dimensionality, i.e., states that live in a finite-
dimensional Hilbert space [8]. A prime example of this are high-dimensional
maximally entangled states, which facilitate entanglement-based quantum
communication protocols such as high-dimensional device-independent quantum
key distribution [3]. Moreover, the ability to programmatically reconfigure
the dimensionality of the generated states can enhance the performance of a
wide range of QIS applications, where different protocols and receivers may
demand specific dimensionalities for efficient operation. However, current
implementations of high-dimensional TM sources typically meet only one of
these essential requirements. In dispersion-engineered parametric-down-
conversion (PDC) sources [15, 16, 17], high-dimensional entangled states are
generated by precisely tailoring the phase-matching function of nonlinear
crystals, preventing the dimensionality of the system from being modified
after the fabrication process. Gas-filled photonic crystal fibers are
nonlinear materials whose dispersion properties can be altered even after
fabrication, allowing for flexible dimensionality control of the generated
entangled states [18]; however, in these materials, the dispersion control is
not accurate enough to guarantee a well-defined dimensionality.

Alternative reconfigurable schemes rely on spectral manipulation or filtering
of broadband frequency-entangled photons applied through pulse shapers [19,
20], Hong-Ou-Mandel interference [21], or Fabry-Pérot cavities after [22, 23,
24] or around [25] the PDC source. These methods generate discrete frequency-
bin-entangled states that can be scaled to higher dimensions by increasing the
spectral bandwidth of the PDC source; however, these states are not maximally
entangled due to the presence of residual frequency anti-correlations [26] or
uneven population of the frequency bins.

Here, we demonstrate a programmable source of high-dimensional maximally
entangled TM states (Fig. 1). Our source is based on a simple setup consisting
of an initially spectrally decorrelated PDC process driven by a spectrally
shaped pump pulse. Shaping the pump pulse into so-called cosine-kernel
functions [27] facilitates the generation of maximally entangled TM states
with well-defined dimensionality within a fixed spectral bandwidth. Notably,
pulse shaping does not require an overhaul of the experimental setup for
accessing different dimensions. The effective dimensionality of the generated
states is characterized via measurements of second-order correlation functions
$g^{(2)}$, which link photon number statistics and modal properties [28]. The
modal structure is additionally analyzed via joint spectral intensities that
directly probe the spectral distribution of the generated photon pairs,
combined with the a priori knowledge of the pump phase. We demonstrate the
generation of bi-photon TM states with a controlled dimensionality starting
from a fully decorrelated (unentangled) state and scaling up to a
20-dimensional entangled state.

Figure 1: Working principle of the programmable source of high-dimensional
entangled photons: the user-chosen dimensionality defines the pump spectrum
for the PDC process, which determines the generated entangled state.

##  2 Theoretical background

PDC is a process in which a pump photon is converted into two lower-energy
photons labeled signal and idler. This process must satisfy the requirements
of energy conservation, described by the pump function
$\alpha(\omega_{\mathrm{s}}+\omega_{\mathrm{i}})$ in terms of the signal and
idler frequencies $\omega_{\mathrm{s}}$ and $\omega_{\mathrm{i}}$,
respectively, and momentum conservation, given by the phase-matching function
$\Phi(\omega_{\mathrm{s}},\omega_{\mathrm{i}})$ which is fixed by material
properties. The product of these two functions is the joint spectral amplitude
(JSA)

$f(\omega_{\mathrm{s}},\omega_{\mathrm{i}})=\alpha(\omega_{\mathrm{s}}+\omega_{\mathrm{i}})\Phi(\omega_{\mathrm{s}},\omega_{\mathrm{i}})\,,$
(1)

which describes the correlations between the signal and idler photons as a
function of their frequencies and, therefore, fully characterizes the PDC
state [29] (Fig. 2).

The effective dimensionality of the Hilbert space in which the PDC state lives
can be determined through a Schmidt decomposition of the corresponding JSA
[29]. This process decomposes the JSA into sets of pairwise orthogonal modes

$f(\omega_{\mathrm{s}},\omega_{\mathrm{i}})=\sum_{k=0}^{r-1}\sqrt{\lambda_{k}}\psi^{k}(\omega_{\mathrm{s}})\varphi^{k}(\omega_{\mathrm{i}})\,,$
(2)

which are separable into signal ($\psi^{k}(\omega_{\mathrm{s}})$) and idler
($\varphi^{k}(\omega_{\mathrm{i}})$) complex spectral functions. The Schmidt
coefficients $\sqrt{\lambda_{k}}$ indicate the relative amplitude of each mode
pair and are normalized such that $\sum_{k}\lambda_{k}=1$. The effective
dimensionality of the PDC state is quantified by the Schmidt number
$K=1/(\sum_{k}\lambda_{k}^{2})$, whereas the number of non-zero
$\sqrt{\lambda_{k}}$ coefficients defines the Schmidt rank $r$. A finite value
of $r$ implies that the generated signal and idler photons can be fully
described by a finite-dimensional basis. In an experimental setting, this
property allows one to fully characterize the PDC state through a finite
number of measurements. If $K$=$r$, i.e., the Schmidt coefficients are
uniformly distributed, then the photon pair is maximally entangled and can be
described as
$\ket{\Psi}=\sqrt{\lambda}\sum_{k=0}^{r-1}\ket{\psi^{k}}\ket{\varphi^{k}}$.

A Schmidt number of $K$=1 indicates a single-mode bi-photon state, i.e., a
spectrally decorrelated JSA. This can be realized for instance through
symmetric group-velocity matching in type-II PDC with a phase-matching
function angled at $45\text{\,}\mathrm{\SIUnitSymbolDegree}$ in the signal-
idler frequency space [30]. The positively-correlated phase-matching function
will orthogonally intercept the pump function, which is always oriented at
$-45\text{\,}\mathrm{\SIUnitSymbolDegree}$ due to energy conservation. If the
pump spectrum and phase-matching function are both Gaussian-shaped and have
the same bandwidth, then the resulting JSA will be a fully separable circle in
frequency space (Fig. 2a).

In order to increase the dimensionality of the entangled state, one needs to
modify the distribution of the Schmidt coefficients via appropriately
tailoring the JSA. Eq. 1 entails that this can be achieved by manipulating
either the phase-matching function or the pump spectrum. While both approaches
are feasible and have been demonstrated, only the latter allows one to apply
these changes “on the fly”, i.e., programmatically and without any hardware
modifications [31]. For this reason, we focus on complex spectral shaping of
the pump pulse [32] as the tool to tailor the PDC state.

We note that this technique fundamentally relies on achieving an initially
decorrelated JSA which, in turn, requires an underlying positively-correlated
phase-matching function. Experimentally, this can be realized in periodically
poled potassium titanyl phosphate (ppKTP), which allows one to reach a phase-
matching angle of approximately $49\text{\,}\mathrm{\SIUnitSymbolDegree}$ in
bulk and $60\text{\,}\mathrm{\SIUnitSymbolDegree}$ in waveguides. The
procedure for achieving decorrelation has been detailed in [33, 30, 31] and,
for this work, is described in the Supplementary Material.

One possibility to generate high-dimensional TM-entangled states is to shape
the pump spectrum as Hermite-Gauss functions [10]. However, while this method
allows for a well-defined dimensionality of the PDC state, it does not produce
maximally entangled states beyond a two-dimensional Bell state [8]. Moreover,
the spectral bandwidth spanned by Hermite-Gauss functions grows wider for
increasing orders, posing a practical challenge for realistic implementations
with a limited available bandwidth.

A notable alternative involves shaping the pump spectrum into the so-called
cosine-kernel (CK) modes [27], i.e., Gaussian spectra modulated by a cosine
function, which are the frequency-space representation of a superposition of
Gaussian time bins. A pump pulse shaped as the $n$-th order CK mode generates
an ($n$+1)-dimensional maximally entangled state (Fig. 2). Unlike Hermite-
Gauss modes, the spectral bandwidth spanned by CK modes is constant for all
mode orders, offering a significant advantage for shaping methods similar to
the one in this work, in which the pump spectrum is “carved” from an initial
pulse with a fixed spectral bandwidth. CK modes present finer spectral
features for increasing mode orders; consequently, the highest-order mode that
can be realized experimentally, and thereby the highest achievable
dimensionality, will then depend solely on the resolution of the shaping
system.

Figure 2: Visual representation of the Schmidt decomposition of the JSA of a
PDC state generated by a pump shaped as a Gaussian (top) and second-order CK
(bottom) function. From left to right: pump spectrum
$\alpha(\omega_{\mathrm{p}})$, where $\omega_{\mathrm{p}}$ is the pump
frequency; JSA, equal to the product of the pump function
$\alpha(\omega_{\mathrm{s}}+\omega_{\mathrm{i}})$ (dashed contour) and phase-
matching function $\Phi(\omega_{\mathrm{s}},\omega_{\mathrm{i}})$ (solid
contour); Schmidt modes and correspondent signal and idler functions. We note
that the Schmidt modes in the bottom row have the same Schmidt coefficient
$\sqrt{\lambda}=1/\sqrt{3}$.

To verify the generation of high-dimensional maximally entangled states, one
must characterize the modal structure of the PDC process [29]. As mentioned
above, this can be done by fully characterizing the JSA and directly
performing a Schmidt decomposition to verify that the Schmidt coefficients are
uniformly distributed. From the Schmidt coefficients, one can also calculate
the Schmidt number $K$ to describe the effective dimensionality of the state.
However, existing methods to fully characterize the JSA of PDC states (based
on state tomography [31] or spectral shearing interferometry [34]) are
extremely resource-expensive, as they require an additional nonlinear process
or a complex interferometric setup.

In 2011, A. Christ et al. [28] demonstrated a resource-efficient method to
probe $K$ using the normalized second-order correlation function $g^{(2)}$ of
pulsed quantum light. This approach is suitable for a measurement system in
which the detection window is longer than the pulse duration but shorter than
the pulse period, which is the typical case in ultrafast quantum optics. In
this regime, the value of $g^{(2)}$ indicates the integral of the second-order
correlation function over one pulse. In the low-gain regime, the $g^{(2)}$ is
connected to the Schmidt number $K$ by the simple relation [28]

$g^{(2)}\approx 1+\frac{1}{K}\,.$ (3)

Although $g^{(2)}$ measurements are an established tool to monitor the
effective dimensionality of a PDC state, they are not sufficient to determine
whether the state is maximally entangled, as a non-uniform distribution of
Schmidt coefficients could also yield an integer value of $K$. Therefore, we
look at $g^{(2)}$ measurements in combination with the joint spectral
intensity (JSI) of the state. In contrast to the JSA, the JSI can be
straightforwardly measured with a simple spectrometric setup requiring only a
dispersive medium and time-resolved detection [35]. The JSI lacks the phase
information necessary to correctly estimate the Schmidt coefficients;
nevertheless, we can assume that the _a priori_ known spectral phase applied
to the pump pulse is mapped to the JSA following the energy conservation
condition, i.e., antidiagonally in the signal-idler frequency space. We can
then apply this phase to the square root of the measured JSI to reconstruct
the JSA, and confirm the maximally entangled condition of the PDC state via a
Schmidt decomposition. This leads us to an estimate of the dimensionality of
the system that we can compare to the effective dimensionality calculated from
the $g^{(2)}$ to confirm the generation of a high-dimensional maximally
entangled state.

##  3 Experiment

Figure 3: Schematic of the experimental setup. a) PDC source. The pump beam
(red line) is generated by a Ti:Sapphire laser and is spectrally shaped by a
4-f waveshaper based on a spatial light modulator (SLM). The shaped beam is
coupled into a periodically poled potassium titanyl phosphate (ppKTP)
waveguide. The photon pair generated by the PDC process in the waveguide is
isolated from the remaining pump light by a broad band pass filter (BP1). The
two photons are separated by a polarizing beam splitter (PBS), filtered again
by narrowband filters BP2 and BP3 and then coupled into single-mode fibers. b)
$g^{(2)}$-measurement configuration. One arm $f_{i/s}$ of the source setup is
connected to a 50/50 fiber beam splitter. The photons in each output of the
beam splitter are detected by a combination of a superconducting nanowire
single-photon detectors (SNSPDs) and a time-tagging unit (TT). c) Measurement
configuration for the time of flight (ToF) spectrograph. The frequencies of
signal and idler photons are mapped to delays by dispersive fibers, and
arrival-time correlations are detected by single-photon detectors and a time-
tagging unit.

A schematic of the experiment is shown in Fig. 3a. Ultrashort laser pulses
with a duration of $150\text{\,}\mathrm{f}\mathrm{s}$ are generated by a
Ti:Sapphire laser with a repetition rate of 80 MHz and a spectrum centered at
$\lambda_{\mathrm{p}}=$758.7\text{\,}\mathrm{nm}$$. The beam is directed to a
folded-4-f waveshaper [36] with a resolution of approximately
$10\text{\,}\mathrm{GHz}$ to prepare the pump modes via amplitude and phase
shaping. The pump beam is then coupled into an 8-mm-long periodically poled
potassium titanyl phosphate (ppKTP) waveguide by AdvR Inc. with a poled length
of $1.5\text{\,}\mathrm{mm}$ and a poling period of
$117\text{\,}\mathrm{\SIUnitSymbolMicro m}$. The type-II PDC process inside
the poled waveguide generates pairs of orthogonally polarized photons with
central wavelengths $\lambda_{\mathrm{0,s}}=$1511\text{\,}\mathrm{nm}$$ and
$\lambda_{\mathrm{0,i}}=$1524\text{\,}\mathrm{nm}$$ for signal and idler,
respectively.

A band-pass filter centered at $1538\text{\,}\mathrm{nm}$ with an acceptance
bandwidth of $82\text{\,}\mathrm{nm}$ (Semrock FF01-1538/82) filters out the
remaining pump light and most of the fluorescence light coming from the
waveguide. The photon pair is then separated by a polarizing beam splitter
(PBS) into the vertically polarized signal and the horizontally polarized
idler. To further filter out fluorescence from the waveguide, both arms
contain a narrow band-pass filter with a full-width-half-maximum of
$7\text{\,}\mathrm{nm}$ angle-tuned to match the centers of the signal and
idler spectra. The filter bandwidth is very close to the spectral bandwidth of
the photons (approximately $7\text{\,}\mathrm{nm}$ and $5\text{\,}\mathrm{nm}$
for signal and idler, respectively) as it allows for significantly suppressing
fluorescence noise while transmitting most of the PDC photons. After
filtering, the photons are coupled into single-mode fibers and detected by two
superconducting nanowire single-photon detectors (SNSPDs) with an efficiency
of 80%.

We estimate the optical losses of the setup through the Klyshko efficiency
[37], defined as $\eta_{\mathrm{s(i)}}=p_{\mathrm{coinc}}/p_{\mathrm{i(s)}}$
for the signal (idler) arm, where $p_{\mathrm{coinc}}$ is the probability to
detect a coincidence between both arms and $p_{\mathrm{i(s)}}$ is the
probability to have a click in the idler (signal) arm. Effectively, the
Klyshko efficiency measures the probability of detecting one photon from a
generated photon pair, given that the other photon has been detected. We
measure $\eta_{\mathrm{s}}\approx 25\%$ and $\eta_{\mathrm{i}}\approx 25\%$,
where the optical losses are mostly due to the fiber couplings.

For the $g^{(2)}$ measurements, one of the fiber out-couplings is connected to
a fiber beam splitter, each output of which is detected by an SNSPD (Fig. 3b).
Photon arrival times are recorded by a Swabian Instruments time-tagging unit,
and time filtering is performed to suppress the fluorescence counts. Using the
$g^{(2)}$ value as reference, we optimize the bandwidth and chirp (quadratic
phase coefficient) parameters via the waveshaper to obtain the initial
decorrelated PDC state from a Gaussian-shaped pump (see Supplementary Material
for details).

We measure the JSI using a time-of-flight (ToF) spectrograph [35] (Fig. 3c).
In this configuration, a dispersive fiber with a chromatic dispersion
coefficient of
$-418\text{\,}\mathrm{p}\mathrm{s}\mathrm{/}\mathrm{n}\mathrm{m}$ is added to
each of the PDC output arms to map frequencies to delays. The arrival times of
the photons are then measured by two SNSPDs connected to a time-tagging unit.
The pulse duration is short enough not to affect this measurement. The
correlation between the arrival times of paired signal and idler is used to
reconstruct the JSI.

##  4 Results and discussion

Figure 4: Experimental Schmidt number $K$ for each CK mode $n$ obtained from
$g^{(2)}$ measurements in the signal and idler arms (orange and blue points)
and values reconstructed from the deconvolved measured JSI combined with the
known pump phase (green cross), compared to the theoretical values
$K_{\mathrm{th}}$ (red line). The marked points correspond to the CK modes
analyzed in Fig. 5. Figure 5: For three different CK mode orders $n$: measured
JSI (first row), JSA reconstructed from the deconvolved measured JSI combined
with the known pump phase (second row), and resulting distribution of the
Schmidt weights $\lambda_{k}$ (blue bars) for each Schmidt mode $k$, compared
to the expected weights for a maximally entangled state (red line).

Fig. 4 shows the measured effective dimensionality $K$ of the PDC states as a
function of the programmed CK order $n$ compared to the theoretical values
$K_{\mathrm{th}}$. The experimental dimensionality, characterized via
$g^{(2)}$ measurements of the signal and idler arms separately, grows linearly
with $n$, confirming that the proposed system can successfully allow the user
to programmatically select the effective dimensionality of the PDC state in up
to 20 dimensions.

The initial values for the signal and idler arms,
$g^{(2)}_{0,\mathrm{s}}=1.93\pm 0.01$ and $g^{(2)}_{0,\mathrm{i}}=1.88\pm
0.01$, correspond to a Schmidt number $K_{0,\mathrm{s}}=1.08\pm 0.01$ and
$K_{0,\mathrm{i}}=1.14\pm 0.01$, respectively. These values are slightly
larger than the expected fully decorrelated state value of
$K_{0,\mathrm{th}}=1$, primarily due to non-ideal phase-matching conditions:
the phase-matching function of the ppKTP waveguide, shaped as a sinc at an
angle of $31\text{\,}\mathrm{\SIUnitSymbolDegree}$ with respect to the signal
axis, deviates from the ideal condition of a perfectly diagonal Gaussian
phase-matching. This non-ideal configuration leads to unwanted correlations
between signal and idler frequencies, introducing a small multi-mode component
to the PDC state. While these imperfections could be mitigated with the use of
an apodized waveguide with a Gaussian phase-matching spectrum [38, 39]
engineered to have a perfect positive frequency correlation between signal and
idler, their impact in this source is limited and is further reduced by
spectral filters, which cut out the side lobes of the sinc-shaped phase-
matching function.

Above $K=18$, one can notice a higher discrepancy between the theoretical and
experimental dimensionality. This is a consequence of the limited resolution
of the waveshaper, which fails to perfectly shape the narrow features of high-
order CK functions. As a matter of fact, the CK function of order $n=17$ has a
main peak with a spectral width of approximately $30\text{\,}\mathrm{GHz}$,
only three times larger than the resolution of the waveshaper. This issue can
be easily addressed by using an improved waveshaper: increasing the resolution
by only a factor 2 would allow one to generate CK functions up to order $n=32$
before reaching the shaping limits.

Fig. 5 (a), (b) and (c) show, for three different CK orders, the measured JSI
(first row), the JSA reconstructed by applying the known pump phase to the
measured JSI (second row) and the weights $\lambda_{k}$ obtained from the
Schmidt decomposition of the JSA (last row). We note that the limited
resolution of the ToF spectrograph introduced an artificial broadening of the
fine spectral features of high-order CK spectra; therefore, to eliminate these
artifacts, we performed a deconvolution of the direct JSI measurements using
the measured point-spread function of the ToF setup (see Supplementary
Material for a detailed explanation).

The resulting Schmidt weights distribution (in blue) presents a clear edge at
$k=K_{\mathrm{th}}$, as expected from a maximally entangled state (in red).
The zero value of the Schmidt weights beyond $K_{\mathrm{th}}$ indicates that
the presented source generates states with a finite dimensionality, i.e.,
containing a finite number of Schmidt modes. The main discrepancy with the
theory lies in the uneven value of the coefficients, which we attribute to the
spectral distortions induced by the narrowband filters in each arm. As one can
observe in the square edges in Fig. 5(a), the filters have a slightly narrower
bandwidth than the generated photons and, as a consequence, introduce spectral
distortions in the PDC state, which result in uneven additional optical losses
for different Schmidt modes. Despite this drawback, narrow filtering is
beneficial for $g^{(2)}$ measurements as it limits fluorescence noise, which
would otherwise artificially increase the detected single counts and lead to a
less accurate assessment of the $g^{(2)}$ value of the source. Contrarily,
since JSI measurements are not as significantly affected by fluorescence, the
effect of the induced distortions becomes noticeable. This is especially
evident in the signal arm, which has a wider spectrum than the idler due to
the sub-optimal phase-matching angle. Nevertheless, the unevenness of the
Schmidt weights is small enough not to significantly affect $K$, as indicated
by the Schmidt number $K$ calculated from this distribution (the green cross
in Fig. 4) being very close to the theoretical value $K_{\mathrm{th}}$.
Furthermore, the Schmidt weights can be directly addressed by tailoring the
pump spectrum, allowing one to make their distribution more even if necessary.

We refer the reader to the Supplementary Material for a more detailed analysis
of the causes of experimental imperfections and possible solutions. We note
that the highlighted challenges are technical, rather than fundamental
limitations, and can be readily addressed, e.g., by employing a dispersion-
engineered waveguide, a higher-resolution waveshaper, and spectral filters
with an optimally matched bandwidth.

##  5 Conclusion

We demonstrated the generation of time-frequency-entangled photons with a
programmable dimensionality ranging from 1 (decorrelated state) up to 20
dimensions, selected via straightforward spectral shaping of the pump pulse.
By analyzing the measured joint spectral intensity alongside the a priori
knowledge of the pump phase, we inferred a Schmidt coefficients distribution
compatible with high-dimensional maximally entangled states. While our results
and the theoretical predictions strongly suggest maximal time-frequency
entanglement, a rigorous proof will necessitate a careful phase-sensitive
characterization of the generated PDC states. This can be accomplished through
high-dimensional state tomography, facilitated, e.g., by a so-called multi-
output quantum pulse gate [11].

The performance of the current system is limited solely by technical
constraints, which can be substantially enhanced by incorporating state-of-
the-art experimental components. These improvements will facilitate the
generation of maximally entangled states in more than 40 dimensions. Beyond
dimensional control, manipulating the pump spectrum offers the potential not
only to program the state dimensionality, but also to finely adjust the
Schmidt weights themselves, thereby achieving an unprecedented degree of
entanglement control.

The ongoing development of our source and the integration of cutting-edge
components will pave the way for groundbreaking applications in quantum
information processing, quantum communication, and quantum metrology,
establishing our platform as a promising resource for advancing the
capabilities of quantum technologies.

Funding This research was supported by the EU H2020 QuantERA ERA-NET Cofund in
Quantum Technologies project QuICHE, and by the Deutsche
Forschungsgemeinschaft (DFG, German Research Foundation) – SFB-
Geschäftszeichen TRR142/3-2022 – Projektnummer 231447078; Project C07.

Acknowledgments The authors thank V. Ansari for helpful discussions.

Disclosures The authors declare no conflicts of interest.

Data availability Data underlying the results presented in this paper are not
publicly available at this time but may be obtained from the authors upon
reasonable request.

Supplemental document See Supplement 1 for supporting content.

## References

  * [1] H. J. Kimble, “The quantum internet,” Nature 453, 1023–1030 (2008).
  * [2] Z.-S. Yuan, X.-H. Bao, C.-Y. Lu, _et al._ , “Entangled photons and quantum communication,” Physics Reports 497, 1–40 (2010).
  * [3] L. Sheridan and V. Scarani, “Security proof for quantum key distribution using qudit systems,” Physical Review A 82 (2010).
  * [4] G. C. G. Berkhout, M. P. J. Lavery, J. Courtial, _et al._ , “Efficient sorting of orbital angular momentum states of light,” Phys. Rev. Lett. 105, 153601 (2010).
  * [5] K. Huang, H. Liu, S. Restuccia, _et al._ , “Spiniform phase-encoded metagratings entangling arbitrary rational-order orbital angular momentum,” Light: Science & Applications 7, 17156–17156 (2018).
  * [6] M. Mirhosseini, O. S. Magaña-Loaiza, M. N. O’Sullivan, _et al._ , “High-dimensional quantum cryptography with twisted light,” New Journal of Physics 17, 033033 (2015).
  * [7] D. Cozzolino, B. Da Lio, D. Bacco, and L. K. Oxenløwe, “High-dimensional quantum communication: Benefits, progress, and future challenges,” Advanced Quantum Technologies 2, 1900038 (2019).
  * [8] B. Brecht, D. V. Reddy, C. Silberhorn, and M. G. Raymer, “Photon temporal modes: A complete framework for quantum information science,” Phys. Rev. X 5, 041017 (2015).
  * [9] B. Brecht, A. Eckstein, R. Ricken, _et al._ , “Demonstration of coherent time-frequency schmidt mode selection using dispersion-engineered frequency conversion,” Phys. Rev. A 90, 030302(R) (2014).
  * [10] V. Ansari, E. Roccia, M. Santandrea, _et al._ , “Heralded generation of high-purity ultrashort single photons in programmable temporal shapes,” Opt. Express 26, 2764–2774 (2018).
  * [11] L. Serino, J. Gil-Lopez, M. Stefszky, _et al._ , “Realization of a multi-output quantum pulse gate for decoding high-dimensional temporal modes of single-photon states,” PRX Quantum 4, 020306 (2023).
  * [12] P. C. Humphreys, B. J. Metcalf, J. B. Spring, _et al._ , “Linear optical quantum computing in a single spatial mode,” Phys. Rev. Lett. 111, 150501 (2013).
  * [13] H.-H. Lu, J. M. Lukens, N. A. Peters, _et al._ , “Electro-optic frequency beam splitters and tritters for high-fidelity photonic quantum information processing,” Phys. Rev. Lett. 120, 030502 (2018).
  * [14] H.-H. Lu, J. M. Lukens, N. A. Peters, _et al._ , “Quantum interference and correlation control of frequency-bin qubits,” Optica 5, 1455–1460 (2018).
  * [15] F. Graffitti, P. Barrow, A. Pickston, _et al._ , “Direct generation of tailored pulse-mode entanglement,” Physical Review Letters 124, 1–6 (2020).
  * [16] C. L. Morrison, F. Graffitti, P. Barrow, _et al._ , “Frequency-bin entanglement from domain-engineered down-conversion,” APL Photonics 7, 066102 (2022).
  * [17] F. Chiriano, J. Ho, C. L. Morrison, _et al._ , “Hyper-entanglement between pulse modes and frequency bins,” Opt. Express 31, 35131–35142 (2023).
  * [18] M. A. Finger, N. Y. Joly, P. S. J. Russell, and M. V. Chekhova, “Characterization and shaping of the time-frequency schmidt mode spectrum of bright twin beams generated in gas-filled hollow-core photonic crystal fibers,” Physical Review A 95, 1–10 (2017).
  * [19] C. Bernhard, B. Bessire, T. Feurer, and A. Stefanov, “Shaping frequency-entangled qudits,” Physical Review A - Atomic, Molecular, and Optical Physics 88, 1–4 (2013).
  * [20] N. B. Lingaraju, H.-H. Lu, S. Seshadri, _et al._ , “Quantum frequency combs and Hong–Ou–Mandel interferometry: the role of spectral phase coherence,” Optics Express 27, 38683 (2019).
  * [21] R. B. Jin, R. Shimizu, M. Fujiwara, _et al._ , “Simple method of generating and distributing frequency-entangled qudits,” Quantum Science and Technology 1 (2016).
  * [22] Z. Xie, T. Zhong, S. Shrestha, _et al._ , “Harnessing high-dimensional hyperentanglement through a biphoton frequency comb,” Nature Photonics 9, 536–542 (2015).
  * [23] G. Maltese, M. I. Amanti, F. Appas, _et al._ , “Generation and symmetry control of quantum frequency combs,” npj Quantum Information 6 (2020).
  * [24] K.-C. Chang, X. Cheng, M. C. Sarihan, _et al._ , “648 Hilbert-space dimensionality in a biphoton frequency comb: entanglement of formation and Schmidt mode decomposition,” npj Quantum Information 7, 48 (2021).
  * [25] Z. Y. Ou and Y. J. Lu, “Cavity enhanced spontaneous parametric down-conversion for the prolongation of correlation time between conjugate photons,” Physical Review Letters 83, 2556–2559 (1999).
  * [26] H.-H. Lu, M. Liscidini, A. L. Gaeta, _et al._ , “Frequency-bin photonic quantum information,” Optica 10, 1655 (2023).
  * [27] G. Patera, C. Navarrete-Benlloch, G. J. de Valcárcel, and C. Fabre, “Quantum coherent control of highly multipartite continuous-variable entangled states by tailoring parametric interactions,” The European Physical Journal D 66, 241 (2012).
  * [28] A. Christ, K. Laiho, A. Eckstein, _et al._ , “Probing multimode squeezing with correlation functions,” New Journal of Physics 13, 033027 (2011).
  * [29] C. K. Law, I. A. Walmsley, and J. H. Eberly, “Continuous frequency entanglement: Effective finite hilbert space and entropy control,” Phys. Rev. Lett. 84, 5304–5307 (2000).
  * [30] G. Harder, V. Ansari, B. Brecht, _et al._ , “An optimized photon pair source for quantum circuits,” Opt. Express 21, 13975–13985 (2013).
  * [31] V. Ansari, J. M. Donohue, M. Allgaier, _et al._ , “Tomography and purification of the temporal-mode structure of quantum light,” Phys. Rev. Lett. 120, 213601 (2018).
  * [32] A. M. Weiner, “Femtosecond pulse shaping using spatial light modulators,” Review of Scientific Instruments 71, 1929–1960 (2000).
  * [33] A. Eckstein, A. Christ, P. J. Mosley, and C. Silberhorn, “Highly efficient single-pass source of pulsed single-mode twin beams of light,” Physical Review Letters 106, 1–4 (2011).
  * [34] A. O. C. Davis, V. Thiel, and B. J. Smith, “Measuring the quantum state of a photon pair entangled in frequency and time,” Optica 7, 1317 (2020).
  * [35] M. Avenhaus, A. Eckstein, P. J. Mosley, and C. Silberhorn, “Fiber-assisted single-photon spectrograph,” Opt. Lett. 34, 2873–2875 (2009).
  * [36] A. Monmayrant, S. Weber, and B. Chatel, “Phd tutorial: A newcomer’s guide to ultrashort pulse shaping and characterization,” Journal of Physics B-atomic Molecular and Optical Physics - J PHYS-B-AT MOL OPT PHYS 43 (2010).
  * [37] D. N. Klyshko, “Use of Two-Photon Light for Absolute Calibration of Photoelectric Detectors.” Soviet journal of quantum electronics 7, 1112–1116 (1980).
  * [38] A. Dosseva, L. Cincio, and A. M. Brańczyk, “Shaping the joint spectrum of down-converted photons through optimized custom poling,” Phys. Rev. A 93, 013801 (2016).
  * [39] F. Graffitti, D. Kundys, D. T. Reid, _et al._ , “Pure down-conversion photons through sub-coherence-length domain engineering,” Quantum Science and Technology 2 (2017).


# Automated Annotation with Generative AI Requires Validation

Nicholas Pangakis, Samuel Wolken, and Neil Fasching Department of Political
Science, University of PennsylvaniaDepartment of Political Science and
Annenberg School for Communication, University of PennsylvaniaAnnenberg School
for Communication, University of Pennsylvania

###### Abstract

Generative large language models (LLMs) can be a powerful tool for augmenting
text annotation procedures, but their performance varies across annotation
tasks due to prompt quality, text data idiosyncrasies, and conceptual
difficulty. Because these challenges will persist even as LLM technology
improves, we argue that any automated annotation process using an LLM must
validate the LLM’s performance against labels generated by humans. To this
end, we outline a workflow to harness the annotation potential of LLMs in a
principled, efficient way. Using GPT-4, we validate this approach by
replicating 27 annotation tasks across 11 datasets from recent social science
articles in high-impact journals. We find that LLM performance for text
annotation is promising but highly contingent on both the dataset and the type
of annotation task, which reinforces the necessity to validate on a task-by-
task basis. We make available easy-to-use software designed to implement our
workflow and streamline the deployment of LLMs for automated
annotation.††Author contributions: NP and SW developed the code, analyzed the
data, and wrote the manuscript. NP, SW, and NF collected data and edited the
manuscript.††We thank Yphtach Lelkes for his support and guidance and Daniel
Hopkins for his helpful comments. We also thank any author that sent us their
data.

##  1 Introduction

Many tasks in natural language processing (NLP) depend on high-quality,
manually-labeled text data for training and validation. The manual annotation
process, however, poses non-trivial challenges. In addition to being time
consuming and expensive, human annotators, usually crowdsourced
workers111Researchers have raised concerns about the quality of data collected
from workers on popular crowdsourcing platforms like Amazon’s Mechanical Turk
(Chmielewski and Kucker,, 2020; Douglas et al.,, 2023) or undergraduate
assistants, often suffer from a limited attention span, fatigue, and changing
perceptions of the underlying conceptual categories throughout the labeling
procedures (Grimmer and Stewart,, 2013; Neuendorf,, 2016). When labeling large
amounts of data, these limitations can lead to labeled text data that suffer
from inconsistencies and errors that may be unobservable and
correlated—especially when using coders with similar demographic backgrounds.

To address these challenges, researchers have recently explored the potential
of generative large language models (LLMs), such as ChatGPT, to replace human
annotators. LLMs are faster, cheaper, reproducible, and not susceptible to
some of the pitfalls in human annotation. Prior applications using LLMs in
this capacity, however, have yielded mixed results. Gilardi et al., (2023)
claim that ChatGPT outperforms MTurkers on a variety of annotation tasks.
Reiss, (2023), on the other hand, finds that LLMs perform poorly on text
annotation and argues that this tool should be used cautiously. Numerous other
studies present similar analyses with varying results and recommendations
(e.g., He et al.,, 2023; Wang et al.,, 2021; Ziems et al.,, 2023; Zhu et al.,,
2023; Ding et al.,, 2022).

Although existing research provides valuable insights into both the benefits
and potential constraints of utilizing LLMs for text annotation, it is unclear
whether the automated labeling workflows used in these studies can be
confidently applied to other datasets and tasks, given that they report
minimal performance metrics (i.e., only accuracy or intercoder agreement) and
analyze a few insular tasks on a small number of datasets. Moreover, due to
the rapid development of LLM technology, it is likely that any binary
recommendations about the competence of LLMs at one-off tasks will be quickly
outdated with additional LLM advancements. While several studies so far have
applied LLMs to a broader range of datasets (Ziems et al.,, 2023; Zhu et al.,,
2023), they exclusively test LLM annotation performance on popular, publicly
available benchmark datasets. As such, these tests are plausibly affected by
contamination (Brown et al.,, 2020), meaning that the datasets may be included
in the LLM’s training data and that strong performance may reflect
memorization, which will not generalize to new datasets and tasks. Without
clear guidance on a recommended workflow to use these tools safely and
effectively, academics and practitioners alike could deploy this type of tool
when it has suboptimal performance.

Our primary argument is that a researcher using an LLM for automated
annotation must always validate the LLM’s performance against a subset of
high-quality human-annotated labels.222By high quality, we refer to labels
generated by subject matter experts (not undergraduate assistants or
crowdsourced workers) who carefully label a small subset of data. As we
describe below, our recommended workflow only requires a fraction of the
labeled text data when compared to existing procedures. While LLMs can be an
effective tool to augment the annotation process, there are circumstances
where LLMs fail to deliver accurate results due to factors such as ineffective
prompts, noisy text data, and difficult annotation tasks. Because these
challenges will persist even as LLM technology improves,333For example, even
as LLM technology increases in sophistication, humans could still provide
ambiguous prompt instructions, which could negatively affect LLM performance
on an annotation task. researchers augmenting their text annotation approach
with LLMs must always validate on a task-by-task basis. Rigorous validation
can help researchers craft effective prompts and determine whether LLM
annotation is viable for their classification tasks and datasets. To this end,
we outline a recommended workflow to harness the potential of LLMs for text
annotation in a principled, efficient way.

In this paper, we propose and test a workflow for augmenting and automating
annotation projects with LLMs. We validate our approach and test the
capabilities of LLMs on a wide range of annotation tasks from different, non-
public datasets using appropriate performance metrics. We use LLMs to
replicate 27 different annotation processes from 11 non-public datasets used
from articles recently featured in high-impact publications. In total, we
classified over 200,000 text samples using an LLM (i.e., GPT-4).444Scholars
have raised concerns about relying on LLMs in social science research due to
the constant evolution of LLM software, the black-box nature of how LLMs
process queries, and ambiguity about the training data that underlies LLMs.
Open-source LLMs represent a viable solution to many of these problems
(Spirling,, 2023). The workflow outlined here is LLM-agnostic and could be
adapted to any open-source LLM.

Our findings indicate that LLM performance for text annotation is promising
but highly contingent on both the dataset and the type of annotation task.
Across all tasks, we report a median accuracy of 0.850 and a median F1 of
0.707. Despite the strong overall performance, nine of the 27 tasks had either
precision or recall below 0.5, which reinforces the necessity of researcher
validation. Given the variation in performance across tasks, we identify four
different use cases for automated annotation procedures guided by the LLMs
validation performance. These include using an LLM to check the quality of
human-labeled data, using an LLM to identify cases to prioritize for human
review, using an LLM to produce labeled data to finetune and validate a
supervised classifier, and using an LLM to classify an entire text corpus.

In addition to recommending a standardized process to validate when and how to
use LLMs, we also introduce several novel tools in our automated annotation
procedures.555Code available here: https://github.com/npangakis/gpt_annotate
First, we make available easy-to-use software in Python designed to implement
our methods and streamline the deployment of LLMs for automated annotation.
Second, we show the utility of a consistency score. To measure how
consistently an LLM predicts a particular text sample’s label, we repeatedly
classify each text sample at an LLM temperature of 0.6. Treating the modal
answer as the predicted LLM label, we approximate a degree of “consistency”
across each LLM classification. Because there is a strong correlation between
higher consistency scores and the probability of the classification being
correct, consistency scores are an effective way for researchers to identify
edge cases.

##  2 Workflow and validation

Step 1: Researcher creates task-specific instructions (i.e., a codebook). Step
2: Using codebook, subject matter experts annotate random subset of text
samples. Step 3: Use LLM to annotate a subset of the human-labeled data using
the same codebook. Then, evaluate performance by comparing the LLM labels
against the human labels. Step 4: If low performance, refine codebook to
emphasize incorrect classifications. If necessary, repeat steps 2 and 3 with
updated codebook. Step 5: Using final codebook, test LLM performance on
remaining human-labeled samples. Figure 1: Workflow for augmenting text
annotation with an LLM

In the absence of clear guidance on a recommended workflow for utilizing LLMs
for annotation tasks, researchers run the risk of deploying these tools
despite poor performance. Displayed in Figure 1, we offer a five-step workflow
for incorporating LLMs in a way that foregrounds human judgment, includes an
opportunity for human-in-the-loop refinement of instructions, and gives a
clear indication of LLM performance with minimal ex-ante investment of
resources. We designed this workflow with two motivations: to validate LLM
performance and to refine prompts for LLM classification when possible.

As Figure 1 shows, researchers should first craft a set of task-specific
instructions (i.e., a codebook)666Because LLMs respond to natural language
prompts, a clear codebook is essential for LLM annotation tasks. In general,
an annotation codebook should clearly delineate the concepts of interest. A
rich literature spanning both qualitative and quantitative social science
offers guidance on how to develop a codebook and classify text data based on
relevant concepts (e.g., Krippendorff,, 2018; Crabtree and Miller,, 1992;
MacQueen et al.,, 1998). and then have at least two subject matter experts and
an LLM annotate the same text samples—with the sample size depending on the
type of task and class imbalance.777Based on our analyses, the number of
samples annotated should range between 250 and 1,250 random text samples. For
annotation tasks that involve identifying very rare cases (for instance, cases
in which fewer than 1 percent of text samples are coded as positive cases for
a particular dimension), it may be necessary to label significantly more text
samples by hand. Critically, both the human coders and the LLM should use the
same codebook for annotation, where the codebook serves as the LLM’s prompt
instructions.888If humans and LLMs label the data using different codebooks,
there could be a conceptual gap between the two annotation instructions. Then,
the researcher should evaluate performance (i.e., accuracy, recall, precision,
and F1) by comparing the predicted LLM labels against the human labels. As an
initial step, researchers should first have the LLM annotate a subset of the
human-labeled text samples. If LLM performance is low on this subset,
researchers can refine the codebook instructions by emphasizing incorrect
classifications (repeating this process if necessary).999After updating the
codebook, researchers may want to use the updated codebook to classify the
same subset of cases again to measure the magnitude of change their edits
caused. This exercise is overfitting to a particular subset of data and thus
should not be taken as the barometer of performance. Instead, it offers a
straightforward gauge of how significantly the codebook edits affect LLM
decision-making. If fundamental changes were made to the codebook, the
researcher may want to have the humans re-label the text samples as well.
Finally, using the updated codebook, researchers should test the LLM
performance against the remaining human-labeled samples. Performance on this
held-out data should be used to determine whether an LLM can be effectively
utilized for automated annotation. The code made available in our GitHub
repository offers a simple and efficient way to implement the procedures
outlined in the above workflow.

To validate the proposed workflow, we use an LLM to replicate 27 annotation
tasks across 11 datasets. To ensure these tasks represent a range of
annotation tasks in contemporary social science research, we draw from
research published in outlets across a spectrum of disciplines ranging from
interdisciplinary publications (e.g., Science Advances and Proceedings of the
National Academy of Sciences) to high-impact field journals in political
science (e.g., American Political Science Review and American Journal of
Political Science) and psychology (e.g., Journal of Personality and Social
Psychology). In the Appendix, Table A1 includes the complete list of
replication articles and Table A2 provides brief descriptions of the
annotation tasks in this articles.101010To find these articles, we searched
high-impact journals for articles that implemented some type of manual
annotation procedure. If we were able to acquire the text data, we replicated
all annotation procedures from articles that were published within the last
three years. These annotation tasks cover an extensive range of social science
applications, from identifying whether Cold War-era texts pertained to foreign
affairs or military matters (Schub,, 2022) to analyzing open-ended survey
responses to classify how people conceptualize where beliefs come from
(Cusimano and Goodwin,, 2020).

In each case, we replicate an annotation task using the human-labeled data
from the original study as the ground truth. To avoid the potential for
contamination, we rely exclusively on datasets stored in password-protected
data archives (e.g., Dataverse) or datasets secured through direct outreach to
authors.111111To harmonize this diverse range of annotation tasks into a
common framework for evaluation, we treat every dimension as a separate binary
annotation task. Thus, if an article includes a classification task with three
potential labels, we split the annotation process into three discrete binary
classification tasks. Whenever possible, we begin with the exact codebook used
in the original research design. If this codebook is not available, we either
quote or paraphrase text from the article or supplementary materials that
describes the concepts of interest.121212We do not observe any relationship
between LLM performance and whether or not the direct codebook was available.

Across all 27 tasks, we annotate slightly over 200,000 text samples using
OpenAI’s GPT-4 API. The overall cost was approximately $420 USD. On average, a
dataset with 1,000 text samples took approximately 2–3 hours to complete seven
iterations (see “Consistency scores” section below). Together, the low cost
and relatively rapid speed demonstrate the potential value of LLM-augmented
annotation for many social science text analysis tasks.

##  3 Results

Classification results are shown in Table 1. The results reported here are
based on “held out” text samples (i.e., not the text samples used in the
codebook update process from Step 4 of the workflow). Across the 27 tasks, LLM
classification performance achieved a median F1 score of 0.707. Figure 2 shows
performance on precision and recall for each classification task. As is
apparent in this figure, LLM classification performance is stronger in recall
than precision for 20 of the 27 tasks. On eight of the 27 tasks, the LLM
achieves remarkably strong performance with precision and recall both
exceeding 0.7.

Metric | Minimum | 25th percentile | Mean | Median | 75th percentile | Maximum
---|---|---|---|---|---|---
Accuracy | 0.674 | 0.808 | 0.855 | 0.85 | 0.905 | 0.981
Precision | 0.033 | 0.472 | 0.615 | 0.650 | 0.809 | 0.957
Recall | 0.25 | 0.631 | 0.749 | 0.829 | 0.899 | 0.982
F1 | 0.059 | 0.557 | 0.660 | 0.707 | 0.830 | 0.969
Table 1: LLM classification performance across 27 tasks from 11 datasets.
Figure 2: Precision and recall for each of 27 replicated classification tasks.
Color reflects dataset, such that points sharing the same color are conducted
on the same text data.

Despite the strong overall performance, nine of the 27 tasks had either
precision or recall below 0.5—and three tasks had both precision and recall
below 0.5. Thus, for a full one-third of tasks, the LLM either missed at least
half of the true positive cases, had more false positives than true positives,
or both. As shown in Table 2, the aggregate performance ranged as low as an F1
score of 0.06. Moreover, LLM performance varied substantially across tasks
within a single dataset. In the most extreme example, F1 ranged from 0.259 to
0.811 on two separate tasks within Card et al., (2022), a difference of 0.552.
These results demonstrate the variability of LLM annotation and, accordingly,
underscore the need for task-specific validation.

###  3.1 Consistency scores

By inducing randomness in the LLM through the use of the temperature
hyperparameter and by repeating the annotation task, we can generate an
empirical measure of variance in the label that we deem a “consistency score.”
We recommend that the researcher have the LLM classify each sample at least
three times with a temperature above 0.131313Temperature is an LLM
hyperparameter that indicates the degree of diversity introduced across LLM
responses. It ranges between 0 and 1. For our analyses, we use a temperature
of 0.6 to label each text sample a minimum of seven times with the same
codebook.141414While our choice of 0.6 is arbitrary, we did confirm that
annotating numerous text samples repeatedly returns better results than a
single classification at temperature 0. Future work could test which
temperature setting returns optimal results. As shown in Figure 3, consistency
score is correlated with accuracy, true positive rate, and true negative rate.
Given a vector of classifications, $C$, with length $l$ for a given
classification task, consistency is measured as the proportion of
classifications that match the modal classification
$\big{(}\frac{1}{l}\sum_{i=1}^{l}C_{i}==C_{mode}\big{)}$. Across tasks,
accuracy is 19.4 percentage points higher for text samples labeled with a
consistency score of 1.0 compared with those labeled with a consistency score
less than 1.0. The true positive rate and true negative rate are 16.4
percentage points and 21.4 percentage points higher, respectively, for fully
consistent classifications. Of all the labeled samples, 85.1% have a
consistency score of 1.0. Therefore, consistency scores offer a useful way of
identifying edge cases or more difficult annotations.

Figure 3: Relationship between consistency score and accuracy, TPR, and TNR.

###  3.2 Codebook updates

For each of the 27 classification tasks, we follow the previously outlined
workflow. In steps 3 and 4 of our workflow, we tested LLM classification
performance on a subset of data and then, if relevant, made iterative human-
in-the-loop updates to the codebook to optimize the prompt for LLM
classification performance. This step can be thought of as an application of
“prompt engineering” in which the researchers attempts to identify patterns in
LLM misclassifications and change the codebook to correct any consistent
misperceptions. For each task, we did at most one round of codebook updates.
To measure the effect that codebook updates had on LLM labeling, we re-label
the training data subsets using the final codebooks.

Figure 4 shows the distributions of change in performance metrics after
updating the codebook. This analysis demonstrates whether and how the codebook
update process affects LLM annotation, holding constant the data and
conceptual categories. In most cases, the codebook update process led to
modest improvement in accuracy and F1. Recall decreased in more cases than
improved after codebook updates. Precision, on the other hand, improved in a
majority of cases, driving the improvement in accuracy and F1. Given these
results, the subtleties of prompt construction do not appear to be a
significant lever on performance. Still, although the magnitude of improvement
was generally small, researchers experiencing subpar LLM classification
performance on their text data can use human-in-the-loop codebook refinement
to ensure that their instructions are not to blame.

Figure 4: Change in LLM annotation performance on training data after one
round of codebook updates.

##  4 Use cases

If an LLM achieves satisfactory performance on domain-specific benchmarks for
a given annotation task, there are numerous ways that the LLM could be
integrated into a text analysis project. How researchers augment their project
with an LLM may depend on the LLM performance against the human labeled data,
their budget, the size of the dataset, and the availability of human
annotators. Table 2 displays four potential use cases.

Use case |  Description
---|---
1) Confirming the quality of human-labeled data |  If a researcher has data that is already labeled by human annotators, an LLM may augment the text analysis process by confirming the quality of the human labels. If LLM performance against the human labels is high, this signals that both the LLM and humans made similar annotation decisions. If LLM performance is low, this indicates that either the humans, the LLM, or both made mistakes during the annotation process.
2) Identifying cases to prioritize for human review |  A researcher can manually review annotations with consistency scores lower than 1.0. Additionally, if the particular LLM task achieved high recall, then the researcher may use the LLM to identify potential positive cases in previously unseen data. Then, human annotators can manually review all positive-labeled cases.
3) Producing labeled data to finetune and validate a supervised classifier |  There are various situations in which a researcher may use an LLM to procure training data to finetune a supervised classifier for labeling their corpus.
4) Classifying the entire corpus directly |  In the simplest case, a researcher may find LLM performance to be satisfactory and opt to use the LLM to classify the entire remaining corpus.
Table 2: Use cases for automated annotation with LLMs.

If the LLM’s performance on some or all dimensions is poor, then the
researcher can continue to refine the codebook using new labeled data or
abandon the use of LLMs for the dimensions with unsatisfactory performance. If
researchers opt to continue using an LLM despite initial poor performance, one
strategy to simplify complex annotations may be disaggregation. For instance,
if a researcher finds poor annotation performance on the dimension of “hate
speech,” they may achieve better performance by disaggregating “hate speech”
into component indicators (such as threats, slurs, stereotyping, and so on)
and adding each of the separate indicators as new dimensions into the
codebook, then beginning again at Step 1 in the workflow.

##  5 Conclusion

We observe significant heterogeneity in LLM performance across a range of
social science annotation tasks. Performance depends on both attributes of the
text and the conceptual categories being measured. For instance, whether a
classification task is a factual question about a text sample or requires
judgement has significant bearing on classification strategy (Balagopalan et
al.,, 2023). To address the myriad sources of variation in annotation
performance, we recommend a flexible approach to LLM-augmented annotation that
foregrounds human annotation.

To address these challenges, we present an original workflow for augmenting
text annotation with an LLM along with several use cases for this workflow. We
validate the workflow by replicating 27 annotation tasks taken from 11 social
science articles published in high-impact journals. We find that LLMs can
offer high-quality labels on a wide variety of tasks for a fraction of the
cost and time of alternative options, such as crowd-sourced workers and
undergraduate research assistants. Still, it is imperative that researchers
validate the performance of LLMs on a task-by-task basis, as we find
significant heterogeneity in performance, even across tasks within a single
dataset.

## References

  * Balagopalan et al., (2023) Balagopalan, A., Madras, D., Yang, D. H., Hadfield-Menell, D., Hadfield, G. K., and Ghassemi, M. (2023).  Judging facts, judging norms: Training machine learning models to judge humans requires a modified approach to labeling data.  Science Advances, 9(19):eabq0701.
  * Brown et al., (2020) Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. (2020).  Language models are few-shot learners.  In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.
  * Card et al., (2022) Card, D., Chang, S., Becker, C., Mendelsohn, J., Voigt, R., Boustan, L., Abramitzky, R., and Jurafsky, D. (2022).  Computational analysis of 140 years of us political speeches reveals more positive but increasingly polarized framing of immigration.  Proceedings of the National Academy of Sciences of the United States of America, 31.
  * Chmielewski and Kucker, (2020) Chmielewski, M. and Kucker, S. C. (2020).  An mturk crisis? shifts in data quality and the impact on study results.  Social Psychological and Personality Science, 11(4):464–473.
  * Crabtree and Miller, (1992) Crabtree, B. F. and Miller, W. L. (1992).  A template approach to text analysis: Developing and using codebooks.  In Doing qualitative research. Sage Publications.
  * Cusimano and Goodwin, (2020) Cusimano, C. and Goodwin, G. P. (2020).  People judge others to have more voluntary control over beliefs than they themselves do.  Journal of Personality and Social Psychology, 119.
  * Ding et al., (2022) Ding, B., Qin, C., Liu, L., Bing, L., Joty, S., and Li, B. (2022).  Is gpt-3 a good data annotator?
  * Douglas et al., (2023) Douglas, B. D., Ewell, P. J., and Braue, M. (2023).  Data quality in online human-subjects research: Comparisons between mturk, prolific, cloudresearch, qualtrics, and sona.  PLoS One, 18.
  * Gilardi et al., (2023) Gilardi, F., Alizadeh, M., and Kubli, M. (2023).  Chatgpt outperforms crowd-workers for text-annotation tasks.
  * Grimmer and Stewart, (2013) Grimmer, J. and Stewart, B. M. (2013).  Text as data: The promise and pitfalls of automatic content analysis methods for political texts.  Political Analysis, 21(3):267–297.
  * He et al., (2023) He, X., Lin, Z., Gong, Y., Jin, A.-L., Zhang, H., Lin, C., Jiao, J., Yiu, S. M., Duan, N., and Chen, W. (2023).  Annollm: Making large language models to be better crowdsourced annotators.
  * Kennedy et al., (2022) Kennedy, B., Atari, M., Davani, A. M., Yeh, L., Omrani, A., Kim, Y., Coombs, K., Havaldar, S., Portillo-Wightman, G., Gonzalez, E., Hoover, J., Azatian, A., Hussain, A., Lara, A., Olmos, G., Omary, A., Park, C., Wijaya, C., Wang, X., Zhang, Y., and Dehghani, M. (2022).  Introducing the gab hate corpus: defining and applying hate-based rhetoric to social media posts at scale.  Lang Resources & Evaluation.
  * Krippendorff, (2018) Krippendorff, K. (2018).  Content Analysis: An Introduction to Its Methodology.  Sage, 4 edition.
  * MacQueen et al., (1998) MacQueen, K. M., McLellan, E., Kay, K., and Milstein, B. (1998).  Codebook development for team-based qualitative analysis.  CAM Journal, 10(2):31–36.
  * Neuendorf, (2016) Neuendorf, K. A. (2016).  The Content Analysis Guidebook.  Sage Publications.
  * Reiss, (2023) Reiss, M. (2023).  Testing the reliability of chatgpt for text annotation and classification: A cautionary remark.  Working paper.
  * Saha et al., (2023) Saha, P., Narla, Kalyan, K., and Mukherjee, A. (2023).  On the rise of fear speech in online social media.  Proceedings of the National Academy of Sciences of the United States of America.
  * Schub, (2022) Schub, R. (2022).  Informing the leader: Bureaucracies and international crises.  American Political Science Review, 116.
  * Spirling, (2023) Spirling, A. (2023).  Why open-source generative ai models are an ethical way forward for science.  Nature, 413.
  * Wang et al., (2021) Wang, S., Liu, Y., Xu, Y., Zhu, C., and Zeng, M. (2021).  Want to reduce labeling cost? GPT-3 can help.  In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4195–4205, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Zhu et al., (2023) Zhu, Y., Zhang, P., Haq, E.-U., Hui, P., and Tyson, G. (2023).  Can chatgpt reproduce human-generated labels? a study of social computing tasks.
  * Ziems et al., (2023) Ziems, C., Held, W., Shaikh, O., Chen, J., , Zhang, Z., and Yang, D. (2023).  Can large language models transform computational social science?  Working paper.

## Appendix

Author(s) |  Title |  Journal |  Year
---|---|---|---
Gohdes |  Repression Technology: Internet Accessibility and State Violence |  American Journal of Political Science |  2020
Hopkins, Lelkes, and Wolken |  The Rise of and Demand for Identity-Oriented Media Coverage |  American Journal of Political Science |  (R&R)
Schub |  Informing the Leader: Bureaucracies and International Crises |  American Political Science Review |  2022
Busby, and Gubler, Hawkins |  Framing and blame attribution in populist rhetoric |  Journal of Politics |  2019
Müller |  The Temporal Focus of Campaign Communication |  Journal of Politics |  2021
Cusimano and Goodwin |  People judge others to have more voluntary control over beliefs than they themselves do |  Journal of Personality and Social Psychology |  2020
Yu and Zhang |  The Impact of Social Identity Conflict on Planning Horizons |  Journal of Personality and Social Psychology |  2022
Card et al. |  Computational analysis of 140 years of US political speeches reveals more positive but increasingly polarized framing of immigration |  PNAS |  2022
Peng, Romero, and Horvat |  Dynamics of cross-platform attention to retracted papers |  PNAS |  2022
Saha et al. |  On the rise of fear speech in online social media 151515This article uses the Gab hate speech corpus (Kennedy et al.,, 2022). We include Saha et al., (2023) here, rather than the original source of the labeled data, to emphasize the application of these data in applied social science research. |  PNAS |  2022
Wojcieszak et al. |  Most users do not follow political elites on Twitter; those who do show overwhelming preferences for ideological congruity |  Science Advances |  2022
Table A1: Sources of annotation tasks that replicated in analysis. Study | Annotation tasks
---|---
Gohdes (2020) |  Code Syrian death records for specific type of killing: targeted or untargeted
Hopkins, Lelkes, & Wolken (2023) |  Coding headlines, tweets, and Facebook share blurbs to identify references to social groups defined by a) race/ethnicity; b) gender/sexuality; c) politics; d) religion
Schub (2020) |  Code presidential-level deliberation texts from the Cold War as political or military
Busby, Gubler, & Hawkins (2019) |  Code open-ended responses for three rhetorical elements: attribution of blame to a specific actor, the attribution of blame to a nefarious elite actor, and a positive mention of the collective people
Müller (2021) |  Code sentences from party manifestos for temporal direction: past, present, or future
Cusimano & Goodwin (2020) |  Code respondents’ written statements on climate change for the presence of either (a) generic reasoning about beliefs or (b) supporting evidence for the belief
Yu & Zhang (2023) |  Code respondents’ plans for the future into two categories:proximate future and distant future
Card et al. (2022) |  Code congressional speeches for whether they are about immigration, along with an accompanying tone: proimmigration, antiimmigration, or neutral
Peng, Romero, & Horvat (2022) |  Code whether tweets express criticism with respect to the findings of academic papers
Saha et al. (2020) |  Code Gab posts as a) fear speech, b) hate speech, or c) normal. Further, a post could have both fear and hate components, and,thus, these were annotated with multiple labels
Wojcieszak et al. (2020) |  Code whether a quote tweet was negative, neutral or positive toward the message and/or the political actor, independently of the tone of the original message
Table A2: Descriptions of annotation tasks replicated in analysis.