# Large Language Models are Few-Shot Training Example Generators:
A Case Study in Fallacy Recognition

Tariq Alhindi1 Smaranda Muresan2 Preslav Nakov1
1Mohamed bin Zayed University of Artificial Intelligence, UAE
2Columbia University, USA
{tariq.alhindi<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Recognizing fallacies is crucial for ensuring the quality and validity of
arguments across various domains. However, computational fallacy recognition
faces challenges due to the diverse genres, domains, and types of fallacies
found in datasets. This leads to a highly multi-class, and even multi-label,
setup with substantial class imbalance. In this study, we aim to enhance
existing models for fallacy recognition by incorporating additional context
and by leveraging large language models to generate synthetic data, thus
increasing the representation of the infrequent classes. We experiment with
GPT3.5 to generate synthetic examples and we examine the impact of prompt
settings for this. Moreover, we explore zero-shot and few-shot scenarios to
evaluate the effectiveness of using the generated examples for training
smaller models within a unified fallacy recognition framework. Furthermore, we
analyze the overlap between the synthetic data and existing fallacy datasets.
Finally, we investigate the usefulness of providing supplementary context for
detecting fallacy types that need such context, e.g., diversion fallacies. Our
evaluation results demonstrate consistent improvements across fallacy types,
datasets, and generators.

##  1 Introduction

Fallacies are common errors in reasoning that can mislead and invalidate
arguments. The capacity to discern fallacies is fundamental to sustaining the
robustness and authenticity of arguments across various domains, such as
public policy, legal reasoning, and scientific discourse Bailin and Battersby
(2016). In recent years, the task of automated fallacy recognition has
attracted significant interest from researchers in the fields of Natural
Language Processing (NLP) and Artificial Intelligence (AI) Amgoud and Besnard
(2013); Hamblin (2022); Goffredo et al. (2022); Alhindi et al. (2022); Jin et
al. (2022). Nevertheless, numerous challenges persist, including the
multiplicity of genres, domains, and fallacy types, which contribute to a
complex multi-class and multi-label task structure compounded by class
imbalances in datasets.

Current state-of-the-art models struggle with the recognition of
underrepresented fallacies, which may often require additional context for
accurate identification, such as diversion fallacies Walton (1996). Moreover,
the variety of fallacies coupled with the broad range of contexts in which
they may occur necessitates a comprehensive and diverse dataset for training
these models. One strategy to combat the challenge of sparse and imbalanced
data in machine learning is data augmentation Wang et al. (2017) by creating
synthetic examples, thereby enhancing the dataset size and diversity and
improving the performance of the machine learning model. Recent advancements
in large language models like GPT-3,3.5,4 Brown et al. (2020) provide a
promising avenue for generating high-quality synthetic examples for fallacy
recognition.

Figure 1: Data Augmentation and Model Training Pipeline

The existing work in computational models for fallacy recognition is still in
its early stages, with limited datasets available. These datasets cover
different types of fallacies in various contexts, such as question and answer
dialog moves, name-calling in social media debates, logical fallacies from
educational websites, and fallacies related to Covid-19 misinformation in
social media and news. Previous work has focused on detecting fallacies in
individual datasets, using techniques like fine-tuning transformers for
sequence tagging Goffredo et al. (2022), and training structure-aware
classifiers Jin et al. (2022). However, fallacy recognition is challenging due
to the high number of classification labels, class imbalance in datasets,
limited dataset sizes, and poor out-of-distribution generalization. Alhindi et
al. (2022) proposes a multitask framework using T5, which converts fallacy
types into natural language instructions, and thus approaches the differences
between fallacy datasets as different tasks, but their approach does not
detect infrequent classes effectively. Goffredo et al. (2022) incorporate
argumentation features to detect fallacies in political debates, while Jin et
al. (2022) trains a structure-aware classifier on fallacies from educational
websites; however, they both focus on a single fallacy scheme from one dataset
while we include multiple fallacy schemes.

We extend previous work on generic fallacy recognition by exploring the
capabilities of large language models to generate synthetic data that augments
manually labeled datasets. We study the effect of the data generated under
zero-shot and few-shot conditions on the downstream task of fallacy
recognition. We also analyze the quality of the synthetic data and its
similarity to the fallacy datasets. Figure 1 shows an overview of our approach
of using GPT3.5 to generate additional training examples in zero/few-shot
settings, then training a T5 model Raffel et al. (2020) for fallacy
recognition on a combination of the original and the synthetic data.

In Section 2, we describe the fallacy datasets included in this work. Then, we
present our synthetic data generation approach in Section 3 and experimental
setup in Section 4. We show the results in Section 5 and analysis of the
similarity between original and synthetic data in Section 6. We then present
an overview of related work and conclude in Sections 7 and 8, respectively.

##  2 Fallacy Datasets

We experiment with the five fallacy datasets covered by Alhindi et al. (2022).
They include fallacies in question-answer pairs in game settings (Argotario)
Habernal et al. (2017), propaganda techniques in news articles (Propaganda) Da
San Martino et al. (2019), logical fallacies from educational websites (Logic)
Jin et al. (2022), and fallacies in misinformation around covid in social
media (Covid) Musi et al. (2022) and climate change news articles (Climate)
Alhindi et al. (2022). These datasets identify different fallacy types and
range from 5 to 18 fallacies. Alhindi et al. (2022) unified the fallacy types
from the four schemes and introduced 28 fallacy types in one unified scheme.
These dataset are different in size as they go from a few hundred examples
(450-880) for Climate, Covid and Argotario, to a few thousands (4,500 to
5,100) for Logic and Propaganda. The total number of examples per fallacy type
varies significantly as it ranges from less than 100 examples for some
fallacies (e.g., False Analogy, Strawman, Whataboutism) to more than 1,000
examples (e.g., Hasty Generalization, Name Calling or Labeling, Loaded
Language). Detailed numbers for each fallacy per dataset and split can be
found in Alhindi et al. (2022).

One main challenge in these datasets is the high imbalance frequency of
classes in a high multi-class, and even multi-label task. The unified model
presented by Alhindi et al. (2022) improves the overall results but still
performs much better on more frequent classes, thus we utilize data
augmentation to address this challenge. In addition, two of the five fallacy
datasets: Propaganda and Climate, are from news articles where the fallacy is
annotated at the sentence or fragment level. Therefore, we study the benefit
from providing additional context to the fallacious segment (sentence or
fragment) by including the preceding or succeeding sentence when available.

##  3 Synthetic Data Generation

To generate additional examples for infrequent fallacy classes, we leverage
gpt-3.5-turbo (henceforth called GPT3.5), a conversational language model, as
a data augmentation tool. We explore zero-shot, 1-shot, 2-shot, and 5-shot
settings to generate examples that have not been seen in the original training
data. These generated examples provide diversity and help address the data
scarcity issue for less frequent fallacy types.

In order to understand the capabilities of pretrained large language models
such as GPT3.5 in producing synthetic data, we control the information
provided in the prompts as follows: i) zeroshot prompts that have no fallacy
example and ask the model to generate an example in one form (e.g. sentence,
tweet, question-answer pair) for a certain fallacy provided in the prompt; ii)
fewshot prompts that list the fallacy type and output form in addition to
providing 1 to 5 examples for the given fallacy type in the prompt. The model
is asked to generate the same number of examples given in the prompt (i.e.
1-shot prompt asks the model to generate 1 example, 5-shot ask for 5 new
examples and so on); iii) fewshot-context prompts that provide the examples of
fallacy and their wider context if available (previous and next sentence) and
asks the language model to do the same by generating both examples for a
certain fallacy and their contexts. Figure 2 shows an example of the 1-shot-
context prompt of the Irrelevant Authority fallacy from the Propaganda
dataset.

Prompt
---
Your task is to perform the following actions:
1\. Read the sentence that has the fallacy of  Irrelevant Authority and its
provided context. The sentence and context below are both delimited by <>.
2\. Generate a similar example of one sentence and its context of the
Irrelevant Authority fallacy. The context should be a bigger chunk of text
that includes the sentence, similar to the provided example sentence and
context below.
Your generated output should be in JSON format with the following keys:
generated_sentence, generated_context.
Sentence: <Carlson cited Dr. Robert Epstein who has said, in Carlson’s words,
“Google alone could determine the outcome of almost any election just by
altering its search selections and we would never know it.”>
Context: <Carlson cited Dr. Robert Epstein who has said, in Carlson’s words,
“Google alone could determine the outcome of almost any election just by
altering its search selections and we would never know it.”. Dr. Robert
Epstein is a social scientist and an expert on Google. >
Generated Output
{
’generated_sentence’: ’As a dog groomer, I can tell you that the earth is
flat.’,
’generated_context’: ’While most people accept that the earth is round, one
dog groomer is convinced otherwise. As a dog groomer, I can tell you that the
earth is flat. Dogs are great navigators, and they always sense when the
ground is flat or sloping.’
}
Figure 2: Example of 1-shot context prompt from the Propaganda dataset of a
sentence that has the Irrelevant Authority fallacy and its context.

For all data augmentation settings, we generate the same number of examples
per fallacy and thus study the quality of the synthetic data using different
prompts for generation. The aim is to address data imbalance in these datasets
so we exclude fallacies that are very frequent, especially if they only exist
in one fallacy scheme and thus have a high number of examples with low
diversity. Following this criteria, we exclude Loaded Language and Name
Calling or Labeling that only appear in Propaganda. We also do not generate
examples for Hasty Generalization in a form similar to the one found in the
Logic dataset, but we generate ones in Covid-19 and climate change domains
since their respective datasets has this fallacy but in very low counts. For
all generated fallacies we double the number of examples with respect to the
number of original examples for a certain fallacy thus maintain comparable
ratios of both original and synthetic data. Also, we cap the number of
synthetic examples for each fallacy to 100 examples generated from each
dataset thus changing the distribution of the training set to bring the very
infrequent classes closer to the overall average number of examples per class.

##  4 Experimental Setup

Similar to Alhindi et al. (2022), we use the T5 model Raffel et al. (2020), a
versatile text-to-text transformer, as the backbone for fallacy recognition by
fine-tuning instruction-based prompts on all fallacy datasets. The prompts are
designed to provide explicit instructions on identifying specific fallacies,
enabling targeted learning within the model. This approach is inline with a
large body of research that utilizes instruction-tuning of large language
models on many tasks Wei et al. (2021); Sanh et al. (2022).

We evaluate the proposed approach on the five fallacy datasets. We train the
T5 model using a combination of the original labeled data and the generated
examples from GPT3.5. We compare the performance of the model under different
settings, including zero-shot, 1-shot, 2-shot, and 5-shot scenarios, with and
without additional context to understand the impact of prompt and data
availability on fallacy recognition.

All fallacy examples, original and synthetic, are transformed into
instruction-based prompts that are used to fine-tune the T5-3 Billion model
(henceforth T53B) in a multitask fashion. The model and hyperparameters are
fixed and we only change the training data that is fed into the model with the
aim to study the ability of a smaller size model such as T53B to learn from
manually annotated or crowdsourced data as well as synthetically generated
data from a larger size model such as GPT3.5. We show the results of all
training conditions for the Propaganda and Climate datasets in Table 1 and for
the Argotario, Logic, and Covid datasets in Table 2. In both tables, we report
the overall accuracy and macro F1 scores for each dataset as well as the F1
scores for each fallacy class. The results cover nine training conditions
where we train on the original training set only (baseline-N (no-context))
similar to Alhindi et al. (2022), and baseline-C with context for datasets
that are from news articles. This applies to the Propaganda and Climate where
this context is available. The remaining seven training conditions all use a
different form of data augmentation depending on the number of examples
provided in the prompt during synthetic data generation which includes zero,
one, two, and five examples. All data augmentations experiments are done with
context (C columns) and no-context (N columns), except zero-shot prompts that
were not enough for GPT3.5 to provide usable examples with contexts in most
cases without providing at least one example in the prompt for GPT3.5 to
follow. Therefore zero-shot prompts are only reported in no-context settings.
We discuss below the effect of data augmentation and adding context in more
details.

##  5 Results

|  |  |  | Data Augmentation
---|---|---|---|---
Dataset | Fallacy | baseline | zero-shot | 1-shot | 2-shot | 5-shot
|  | N | C | N | N | C | N | C | N | C
Propaganda | Black and White Fallacy | .14 | .34 | .39 | .39 | .29 | .35 | .33 | .36 | .34
| Causal Oversimplification | .34 | .27 | .41 | .48 | .27 | .39 | .23 | .44 | .29
| Doubt | .61 | .66 | .67 | .66 | .69 | .66 | .71 | .69 | .68
| Exaggerate/Minimization | .34 | .32 | .44 | .58 | .55 | .58 | .47 | .58 | .49
| Fear or Prejudice | .49 | .44 | .49 | .54 | .49 | .67 | .46 | .51 | .50
| Flag-Waving | .64 | .67 | .67 | .68 | .67 | .67 | .67 | .69 | .69
| Irrelevant Authority | .26 | .30 | .44 | .46 | .44 | .47 | .40 | .41 | .38
| Loaded Language | .79 | .76 | .81 | .83 | .81 | .83 | .79 | .83 | .80
| Name Calling, Labeling | .83 | .79 | .83 | .85 | .82 | .85 | .81 | .86 | .81
| Red Herring | 0 | 0 | 0 | .29 | .22 | 0 | 0 | 0 | 0
| Reductio Ad Hitlerum | .17 | .18 | .29 | .40 | .27 | .44 | .25 | .40 | .22
| Slogans | .49 | .45 | .59 | .56 | .52 | .55 | .51 | .67 | .48
| Strawman | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
| Thought-Termin. Cliches | .29 | .34 | .29 | .50 | .40 | .36 | .41 | .38 | .39
| Whataboutism | 0 | 0 | .29 | .63 | .62 | .53 | .53 | .48 | .47
| Accuracy | .68 | .67 | .71 | .74 | .71 | .73 | .70 | .74 | .70
| Macro | .36 | .37 | .44 | .52 | .47 | .48 | .44 | .49 | .44
Climate | Causal Oversimplification | .35 | .33 | .40 | .53 | .32 | .42 | .30 | .60 | .37
| Cherry Picking | .44 | .41 | .43 | .48 | .44 | .43 | .41 | .46 | .45
| Evading Burden of Proof | 0 | 0 | 0 | .17 | .12 | 0 | .10 | 0 | 0
| False Analogy | 0 | 0 | .36 | .62 | .18 | .35 | .17 | .43 | .17
| Hasty Generalization | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
| Irrelevant Authority | .22 | .25 | .31 | .31 | .31 | .31 | .31 | .43 | .33
| Red Herring | 0 | 0 | .12 | .11 | 0 | .18 | 0 | .18 | 0
| Strawman | .22 | 0 | .40 | .40 | .40 | .36 | .50 | .55 | .40
| Vagueness | .37 | .39 | .34 | .40 | .29 | .36 | .36 | .36 | .24
| Accuracy | .30 | .28 | .34 | .40 | .29 | .34 | .29 | .39 | .30
| Macro | .18 | .15 | .26 | .33 | .23 | .27 | .24 | .33 | .22
Table 1: F1 scores on the Propaganda and Climate datasets using multitask training of T53B model. N: no context to the fallacious segment added. C: context of previous and next sentence to the fallacious segment provided. |  |  |  | Data Augmentation
---|---|---|---|---
Dataset | Fallacy | baseline | zero-shot | 1-shot | 2-shot | 5-shot
|  | N | C | N | N | C | N | C | N | C
Argotario | Ad Hominem | .59 | .63 | .63 | .71 | .64 | .62 | .64 | .65 | .63
| Emotional Language | .64 | .68 | .70 | .71 | .70 | .67 | .69 | .60 | .65
| Hasty Generalization | .46 | .44 | .51 | .47 | .54 | .47 | .52 | .55 | .49
| Irrelevant Authority | .71 | .72 | .80 | .75 | .78 | .74 | .77 | .75 | .78
| Red Herring | .32 | .42 | .47 | .50 | .46 | .44 | .51 | .53 | .52
| Accuracy | .56 | .57 | .61 | .61 | .60 | .59 | .63 | .61 | .62
| Macro | .54 | .58 | .62 | .61 | .61 | .58 | .63 | .61 | .61
Logic | Ad Hominem | .77 | .81 | .87 | .88 | .88 | .88 | .85 | .86 | .88
| Ad Populum | .81 | .80 | .82 | .89 | .87 | .86 | .86 | .89 | .85
| Black and White Fallacy | .84 | .84 | .91 | .91 | .89 | .92 | .89 | .91 | .92
| Causal Oversimplification | .65 | .70 | .81 | .79 | .82 | .81 | .80 | .78 | .79
| Circular Reasoning | .57 | .56 | .68 | .84 | .84 | .80 | .77 | .76 | .83
| Deductive Fallacy | .32 | .29 | .48 | .69 | .57 | .57 | .54 | .56 | .57
| Emotional Language | .55 | .53 | .65 | .76 | .77 | .72 | .74 | .71 | .68
| Equivocation | .22 | 0 | .27 | .57 | .43 | .55 | .39 | .43 | .42
| Fallacy of Extension | .08 | .04 | .48 | .68 | .68 | .64 | .60 | .58 | .64
| Hasty Generalization | .64 | .63 | .72 | .80 | .75 | .77 | .75 | .77 | .79
| Intentional Fallacy | .09 | .15 | .16 | .55 | .48 | .46 | .33 | .35 | .33
| Irrelevant Authority | .56 | .54 | .61 | .74 | .68 | .68 | .72 | .68 | .66
| Red Herring | .24 | .30 | .58 | .78 | .67 | .67 | .61 | .65 | .62
| Accuracy | .58 | .58 | .68 | .79 | .76 | .75 | .73 | .73 | .74
| Macro | .45 | .48 | .62 | .76 | .72 | .72 | .68 | .69 | .69
Covid | Causal Oversimplification | .45 | .53 | .40 | .56 | .59 | .53 | .53 | .50 | .50
| Cherry Picking | .35 | .37 | .37 | .31 | .36 | .28 | .34 | .38 | .38
| Evading Burden of Proof | 0 | 0 | .31 | .45 | .53 | .46 | .57 | .49 | .40
| False Analogy | .33 | .50 | .25 | .29 | .29 | .25 | .29 | .29 | .25
| Hasty Generalization | .17 | 0 | .11 | .17 | .16 | .11 | .25 | .10 | .11
| Irrelevant Authority | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
| Red Herring | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
| Strawman | 0 | 0 | 0 | .20 | 0 | 0 | .17 | .17 | .15
| Vagueness | .09 | 0 | .09 | .27 | .33 | .22 | .30 | .15 | .19
| Accuracy | .23 | .25 | .26 | .30 | .34 | .27 | .36 | .31 | .30
| Macro | .16 | .15 | .17 | .25 | .25 | .20 | .27 | .23 | .22
Table 2: F1 scores on the Argotario, Logic and Covid datasets. N: no
additional context provided. C: context of previous and next sentence provided
where available (Propaganda and Climate only and added with no-context
training sets of the three datasets shown).

###  5.1 Data Augmentation

#### Overall Results.

Adding synthetic data to the original dataset improves the results over the
baselines where only the original training data is used regardless of the data
augmentation method. This is true for both the overall accuracy and macro-F1
scores in all five datasets as shown in Tables 1 and 2 whether the context is
provided or not. Interestingly, 1-shot prompts seem to yield the best results
when compared to both zero-shot and other few-shot settings. This results is
counter to what we initially expected. We hypothesized that 5-shot prompts
that have five examples of a fallacy and ask GPT3.5 to generate five similar
examples to yield synthetic data that is more generic to the fallacy (the one
factor that is common among the five examples in the prompt), and therefore
would help train a model for fallacy recognition to be more resilient. The
first part of our hypothesis seem to be correct i.e. the synthetic examples
are less similar than the ones provided in the prompt as we explore in detail
in Section 6. However, having less similar examples to the training data does
not help the model perform better on these benchmark test sets for fallacy
recognition.

#### Per-Class Results.

Some fallacies show massive gains after data augmentation compared to others.
This is true in the Logic dataset where the Equivocation and Fallacy of
Extension are among fallacies with the biggest gains over baselines. These two
fallacies are also the least frequent in the Logic dataset and thus the impact
of data augmentation is bigger. The diversion fallacies in Propaganda e.g.,
Red Herring, Strawman, Whataboutism are particularly challenging in baseline
settings due to their low counts and complexity since they typically require
external information to the fallacious segment to be properly recognized,
which is especially the case for Strawman where all models fail to make any
correct prediction with or without data augmentation. However, for the other
two (Red Herring and Whataboutism), significant gains are observed with data
augmentation particularly for Whataboutism in 1-shot settings where the
f1-scores jumps to 0.63 compared to 0 in the baseline models.

Some fallacy types present a different level of challenge across datasets due
to their format in a particular dataset, frequency, and the fallacies they are
listed with in the prompt at inference time. For example, Red Herring is
easier to detect in Argotario and Logic in the baseline model to begin with
due to a lower number of fallacies in Argotario, and lack of other diversion
fallacies in those two scheme thus making them more distinct than the other
fallacies and easier to distinguish. However, for Propaganda and Climate, the
baselines get 0 f1-scores for Red Herring and data augmentation helps in
improving the results to 0.29 in 1-shot for Propaganda and 0.18 in 2-shot and
5-shot settings in Climate. Some fallacy types remain challenging to detect
with any kind of data augmentation, such as Strawman in Propaganda, and Hasty
Generalization in Climate given their low counts in the test set (e.g., 2-5
examples) and therefore the test sets might have one particular form of this
fallacy rather than represent the fallacy type in general. Having a train-test
split that can truly evaluate the performance of machine learning models for
this task is not trivial due to the high number of classes and the severe data
imbalance.

Figure 3: Average BLEURT score (y-axis) between original and synthetic data
for each fallacy type in few-shot prompts (x-axis: 1-shot, 2shot, 5-shot).

###  5.2 Effect of Additional Context

The use of context during training is different for Propaganda and Climate in
Table 1 compared to the other three datasets shown in Table 2. The difference
between each N and C columns in Table 1 is rather than only providing a
fallacious segment, we provide a wider context window of the previous and next
sentence when available for the two datasets listed in the table. However,
there is no difference in the training data between the N and C columns for
the three datasets listed in Table 2 i.e. Argotario, Logic, and Covid. The
only difference is that they are combined in the multitask training model with
two other datasets (Propaganda and Climate) where the context is provided.
Since the training is done on all datasets combined with some overlap between
fallacy types across datasets, we report the results on the Argotario, Logic,
and Covid datasets for context-based experiments on all five datasets.

With minor exceptions (e.g. Doubt in Propaganda, Vagueness and overall scores
in Covid), adding context does not improve the results for fallacy
recognition. This could be related to the fact that some fallacy types require
different context than others. For example, Cherry Picking requires
understanding of the trend and Strawman requires the retrieval of the original
argument, while Evading the Burden of Proof needs information regarding the
structure of the argument to assess its validity Goffredo et al. (2022);
Alhindi et al. (2021). Therefore, a unified form of context across 28 fallacy
types does not have consistent improvement over experiments conducted under
similar conditions.

###  5.3 Overall Observations

If we examine all per-class results, we notice some inconsistency of the
results under similar training settings. However, there are two general
observations that are consistent across all results. First, data augmentation
through large language models helps train smaller models on more data that is
beneficial to fallacy recognition. Second, simple context of previous or next
sentence does not provide valuable insight for this task and thus
customization of the type of context based on the fallacy type is needed. In
the next section, we take a closer look at the generated examples and how
similar they are to the original ones.

##  6 Original and Synthetic Data Similarity

In order to understand the reason for 1-shot prompts to generate synthetic
data that is more beneficial to the task, we analyze the similarity between
the generated data and the original training examples shown at the prompts. We
use BLEURT as our metric to calculate the similarity as it has the most
consistent results with human evaluation Sellam et al. (2020).

###  6.1 Similarity with the Training Sets

We calculate the BLEURT score for each original-synthetic example pair where
the original example is the one included in the prompt to generate the
synthetic example. Thus, we only run this analysis on the 1-shot, 2-shot, and
5-shot prompts. For the 2-shot and 5-shot, we report the average score for a
generated example with respect to all original examples included in the prompt
that generated it. Figure 3 shows average BLEURT scores for each fallacy type
in all five datasets. We notice high similarity scores in Argotartio and Logic
that range between 0.45 and 0.30 and much lower scores for Climate and Covid
that range between 0.18 and 0.30. This shows that it is harder in general to
produce examples that are similar to those that naturally appear in
misinformation found in news and social media in comparison to ones from
dialogue in game settings or educational fallacy websites. However, what is
shared across all datasets is that 1-shot prompts tend to be produce more
similar examples to the ones included in the prompts.

On the other hand using 2-shot prompts generates examples that are both less
similar than 1-shots prompts as well as 5-shot prompts. The regain in
similarity in 5-shot prompts could be due to that including more examples
improves the average similarity score for each generated example compared with
all original examples in the prompt. In other words, each synthetic example
generated by a 2-shot prompt is compared with the two examples in the prompt.
Thus is could be very similar to one and very different from the other, which
will keep the average similarity score at a lower value. However, each
synthetic example from 5-shot prompts is generated after seeing five examples
from the original data for the given fallacy and therefore might not be too
similar to a single one of them but rather comparably similar to all five,
which will result in a higher average similarity.

The drop in similarity scores is more significant in some fallacies than
others such as Hasty Generalization in Climate, and Causal Oversimplification
in both Climate and Covid. Some fallacy types break the general pattern of
having the highest score in 1-shot prompts and the lowest in 2-shot prompts
such as Circular Reasoning in Logic, which tends to have more homogeneous
examples in the original training set and thus average similarity score
increase with the number of shots in the prompt.

##  7 Related Work

With the significant focus on the development of generative large language
models (LLM) in recent years Brown et al. (2020); Chowdhery et al. (2022);
Zhang et al. (2022); Touvron et al. (2023), there has been an increase in the
utilization of these models to annotate data Feng et al. (2021); Chen et al.
(2023); He et al. (2023); Bansal and Sharma (2023); Zhang et al. (2023), or
generate additional data instances that can be added to existing training sets
for various tasks Kumar et al. (2020); Schick and Schütze (2021); Wang et al.
(2023, 2021); Ye et al. (2022); Gao et al. (2022); Sahu et al. (2023).

We build on the line of work that uses language models for generating
additional training data. Our work differs form previous work in the following
aspects: i) we particularly focus on the ability of using synthetic data
generated by language models to address data imbalance challenges, ii) we use
zero-shot and few-shot settings to generate synthetic data but use full-shot
training on a mix of original and synthetic data for the downstream task, and
iii) we tackle a challenging task of fallacy recognition to understand the
gains from using large language models for data augmentation.

##  8 Conclusion

Fallacy recognition remains a challenging problem due to the high number of
classes, severe data imbalance and the need in some cases for external
information to the fallacious segment. To mitigate the effect of data
imbalance, we studied the capabilities of large language models to generate
synthetic data that can be used to train smaller models on a combination of
original and synthetic data for fallacy recognition across multiple tasks. The
main observation is that data augmentation through large language models is
beneficial for this task. However, the conditions under which the data is
generated impacts the quality of the synthetic data significantly. Providing
one example in the prompt (1-shot) for a certain fallacy from the original
data and asking GPT3.5 to generate a similar example results in synthetic data
that is more similar to the original data in comparison to no examples (zero-
shot) or more examples (e.g., 2-shot or 5-shot), and benefits downstream
models in detecting fallacy types in test sets from the same distribution. The
value in having synthetic data that is less similar to the original training
data and possibly more generic to the task needs to be tested on data from
unseen fallacy schemes or domains, which presents a potential avenue for
future work. Overall, large language models show great potential to generate
additional training data for the task of fallacy recognition, which can be
used to train smaller size open-source models for this task.

## References

  * Alhindi et al. (2022) Tariq Alhindi, Tuhin Chakrabarty, Elena Musi, and Smaranda Muresan. 2022.  Multitask instruction-based prompting for fallacy recognition.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 8172–8187, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Alhindi et al. (2021) Tariq Alhindi, Brennan McManus, and Smaranda Muresan. 2021.  What to fact-check: Guiding check-worthy information detection in news articles through argumentative discourse structure.  In _Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue_ , pages 380–391, Singapore and Online. Association for Computational Linguistics.
  * Amgoud and Besnard (2013) Leila Amgoud and Philippe Besnard. 2013.  Logical limits of abstract argumentation frameworks.  _Journal of Applied Non-Classical Logics_ , 23(3):229–267.
  * Bailin and Battersby (2016) Sharon Bailin and Mark Battersby. 2016.  _Reason in the balance: An inquiry approach to critical thinking_.  Hackett Publishing.
  * Bansal and Sharma (2023) Parikshit Bansal and Amit Sharma. 2023.  Large language models as annotators: Enhancing generalization of nlp models at minimal cost.  _arXiv preprint arXiv:2306.15766_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Chen et al. (2023) Maximillian Chen, Alexandros Papangelis, Chenyang Tao, Seokhwan Kim, Andy Rosenbaum, Yang Liu, Zhou Yu, and Dilek Hakkani-Tur. 2023.  PLACES: Prompting language models for social conversation synthesis.  In _Findings of the Association for Computational Linguistics: EACL 2023_ , pages 844–868, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_.
  * Da San Martino et al. (2019) Giovanni Da San Martino, Seunghak Yu, Alberto Barrón-Cedeño, Rostislav Petrov, and Preslav Nakov. 2019.  Fine-grained analysis of propaganda in news article.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5636–5646, Hong Kong, China. Association for Computational Linguistics.
  * Feng et al. (2021) Xiachong Feng, Xiaocheng Feng, Libo Qin, Bing Qin, and Ting Liu. 2021.  Language model as an annotator: Exploring DialoGPT for dialogue summarization.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1479–1491, Online. Association for Computational Linguistics.
  * Gao et al. (2022) Jiahui Gao, Renjie Pi, LIN Yong, Hang Xu, Jiacheng Ye, Zhiyong Wu, WEIZHONG ZHANG, Xiaodan Liang, Zhenguo Li, and Lingpeng Kong. 2022.  Self-guided noise-free data generation for efficient zero-shot learning.  In _The Eleventh International Conference on Learning Representations_.
  * Goffredo et al. (2022) Pierpaolo Goffredo, Shohreh Haddadan, Vorakit Vorakitphan, Elena Cabrio, and Serena Villata. 2022.  Fallacious argument classification in political debates.  In _Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI_ , pages 4143–4149.
  * Habernal et al. (2017) Ivan Habernal, Raffael Hannemann, Christian Pollak, Christopher Klamm, Patrick Pauli, and Iryna Gurevych. 2017.  Argotario: Computational argumentation meets serious games.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 7–12, Copenhagen, Denmark. Association for Computational Linguistics.
  * Hamblin (2022) Charles Leonard Hamblin. 2022.  _Fallacies_.  Advanced Reasoning Forum.
  * He et al. (2023) Xingwei He, Zhenghao Lin, Yeyun Gong, Alex Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, Weizhu Chen, et al. 2023.  Annollm: Making large language models to be better crowdsourced annotators.  _arXiv preprint arXiv:2303.16854_.
  * Jin et al. (2022) Zhijing Jin, Abhinav Lalwani, Tejas Vaidhya, Xiaoyu Shen, Yiwen Ding, Zhiheng Lyu, Mrinmaya Sachan, Rada Mihalcea, and Bernhard Schoelkopf. 2022.  Logical fallacy detection.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 7180–7198, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Kumar et al. (2020) Varun Kumar, Ashutosh Choudhary, and Eunah Cho. 2020.  Data augmentation using pre-trained transformer models.  In _Proceedings of the 2nd Workshop on Life-long Learning for Spoken Language Systems_ , pages 18–26, Suzhou, China. Association for Computational Linguistics.
  * Musi et al. (2022) Elena Musi, Myrto Aloumpi, Elinor Carmi, Simeon Yates, and Kay O’Halloran. 2022\.  Developing fake news immunity: fallacies as misinformation triggers during the pandemic.  _Online Journal of Communication and Media Technologies_.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21(140):1–67.
  * Sahu et al. (2023) Gaurav Sahu, Olga Vechtomova, Dzmitry Bahdanau, and Issam H Laradji. 2023.  Promptmix: A class boundary augmentation method for large language model distillation.  _arXiv preprint arXiv:2310.14192_.
  * Sanh et al. (2022) Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2022\.  Multitask prompted training enables zero-shot task generalization.  In _The Tenth International Conference on Learning Representations_.
  * Schick and Schütze (2021) Timo Schick and Hinrich Schütze. 2021.  It’s not just size that matters: Small language models are also few-shot learners.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2339–2352, Online. Association for Computational Linguistics.
  * Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020.  BLEURT: Learning robust metrics for text generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7881–7892, Online. Association for Computational Linguistics.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Walton (1996) Douglas N Walton. 1996.  _Argument structure: A pragmatic theory_.  University of Toronto Press Toronto.
  * Wang et al. (2017) Jason Wang, Luis Perez, et al. 2017.  The effectiveness of data augmentation in image classification using deep learning.  _Convolutional Neural Networks Vis. Recognit_ , 11(2017):1–8.
  * Wang et al. (2023) Shuhe Wang, Xiaofei Sun, Xiaoya Li, Rongbin Ouyang, Fei Wu, Tianwei Zhang, Jiwei Li, and Guoyin Wang. 2023.  Gpt-ner: Named entity recognition via large language models.  _arXiv preprint arXiv:2304.10428_.
  * Wang et al. (2021) Zirui Wang, Adams Wei Yu, Orhan Firat, and Yuan Cao. 2021.  Towards zero-label language learning.  _arXiv preprint arXiv:2109.09193_.
  * Wei et al. (2021) Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021.  Finetuned language models are zero-shot learners.  In _International Conference on Learning Representations_.
  * Ye et al. (2022) Jiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu, Jiangtao Feng, Zhiyong Wu, Tao Yu, and Lingpeng Kong. 2022.  ZeroGen: Efficient zero-shot learning via dataset generation.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 11653–11669, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Zhang et al. (2023) Ruoyu Zhang, Yanzeng Li, Yongliang Ma, Ming Zhou, and Lei Zou. 2023.  Llmaaa: Making large language models as active annotators.  _arXiv preprint arXiv:2310.19596_.
  * Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022.  Opt: Open pre-trained transformer language models.  _arXiv preprint arXiv:2205.01068_.


# Mapping Photocathode Quantum Efficiency with Ghost Imaging

K. Kabra<EMAIL_ADDRESS>Department of Physics and Astronomy, UCLA, Los
Angeles, California 90095, USA  Department of Electrical Engineering, UCLA,
Los Angeles, California 90095, USA  S. Li<EMAIL_ADDRESS>SLAC
National Accelerator Laboratory, Menlo Park, California, 94025, USA  F. Cropp
Department of Physics and Astronomy, UCLA, Los Angeles, California 90095, USA
T. J. Lane  SLAC National Accelerator Laboratory, Menlo Park, California,
94025, USA  P. Musumeci  Department of Physics and Astronomy, UCLA, Los
Angeles, California 90095, USA  D. Ratner  SLAC National Accelerator
Laboratory, Menlo Park, California, 94025, USA

###### Abstract

Measuring the quantum efficiency (QE) map of a photocathode injector typically
requires laser scanning, an invasive operation that involves modifying the
injector laser focus and rastering the focused laser spot across the
photocathode surface. Raster scanning interrupts normal operation and takes
considerable time to setup. In this paper, we demonstrate a novel method of
measuring the QE map using a ghost imaging framework that correlates the
injector laser spatial variation over time with the total charge yield. Ghost
imaging enables passive, real-time monitoring of the QE map without manually
modifying the injector laser or interrupting injector operation. We first
demonstrate the method at the UCLA Pegasus photoinjector with the help of a
digital micromirror device (DMD) and a piezoelectric mirror to increase our
control of the overall transverse variance of the illumination profile. The
reconstruction algorithm parameters are fine-tuned using simulations and the
results are validated against the ground truth map acquired using the
traditional rastering method. Finally, we apply the technique to data acquired
parasitically from the LCLS photoinjector, showing the feasibility of this
method to retrieve a QE map without interrupting normal operation.

††preprint: APS/123-QED

##  I Introduction

Modern electron accelerators use photoemission to generate high brightness
electron beams [1, 2, 3]. In this process, an optical drive laser strikes the
surface of a photocathode, emitting electrons due to the photoelectric effect.
The quantum efficiency (QE) map of a photocathode surface determines the
spatial variation of the electron yield for a given incident photon flux. The
QE map of a photocathode is typically not uniform and deteriorates over time
due to experimental conditions. For example, localized hot spots in the drive
laser can burn the surface, which leads to degraded QE in a localized area.
Monitoring the QE map provides useful information for drive laser shaping to
compensate for QE non-uniformities, which is crucial for obtaining the low
emittances required for high-brightness applications [4, 5]. Typically, the QE
map is measured by focusing the drive laser to a small spot size and scanning
it across the cathode surface. In this configuration, the emitted charge at
each location of the focused laser spot maps out the QE, assuming the emitted
charge is below the space charge limit to preserve the linearity of the
measurement. This technique is by nature invasive to normal operation, as it
changes the optical setup in the drive laser to focus the laser beam on the
cathode surface. The resolution is limited by the focused laser spot size.

Another technique that has been successfully applied to measure the QE map [6]
introduces a digital micromirror device (DMD) in the drive laser beam path. In
this case, step size and spot size in the scans can be independently
controlled with minimal optical realignment. Here, the resolution depends by
the size of the micromirrors, the optical (de)magnification of the imaging
system from the DMD to the cathode, and the signal-to-noise ratio of the
charge measurement. Nevertheless, the use of the DMD (which has lower damage
threshold than standard mirrors) significantly limits the amount of laser
power that can be used for normal cathode illumination, and measurement is
still invasive.

In this work, we present a novel method to measure the QE map using classical
ghost imaging. Ghost imaging is an experimental technique that extracts
spatial information from a single-pixel camera (also known as a “bucket”
detector). In classical ghost imaging, an illumination source is typically
split into two arms: one arm going to the sample under analysis and then to
the bucket detector, the other arm reaching a pixelated detector. By
correlating the bucket detector reading, i.e. the total emission from the
sample, and the spatial variation of the illumination, one can reconstruct the
spatial structure of the sample. Implicit in this is the assumption that the
spatial profile of the incident illumination varies shot-to-shot so that
measurements are sufficiently independent. Classical ghost imaging in the
spatial domain has been demonstrated experimentally with various sources of
illumination, including visible light, optical lasers, x-rays, atoms, and
electrons [7, 8, 9, 10].

Ghost imaging offers several advantages over the state-of-the-art raster scan.
For example, simultaneously illuminating multiple pixels (i.e. “multiplexing”)
improves the signal-to-noise ratio when the dominant source of uncertainty is
an overall detector noise independent from the number of pixels in the image
(the so-called Fellgett’s advantage [11]). Furthermore, multiplexing also
enables the use of compressive sensing [12, 13, 14], which can reduce the
number of measurements needed to reconstruct a target for sparse samples.
Finally, and crucially for QE mapping, the ghost imaging framework can use the
intrinsic noise in the illumination pattern of the photocathode drive laser to
implement a passive measurement method that avoids disturbing normal
operation. See [15] for a summary of the advantages of multiplexing.

It is this final advantage, the ability to measure machine parameters
passively, that makes the application of ghost imaging to QE mapping
particularly attractive for user facilities. In this paper we first
demonstrate the idea with a proof-of-principle measurement on a dedicated test
facility, and then show results from a passive QE measurement acquired during
beam delivery at the Linac Coherent Light Source (LCLS).

##  II Classical Ghost Imaging for Cathodes

Ghost imaging requires a varying, known illumination pattern paired with
synchronous measurements of a bucket detector. In typical ghost imaging
experiments, the variation is generated by inserting a random pattern in the
beam path (see e.g. [8]) or by directly controlling the illumination pattern
[16]. For the latter case, one can introduce variation by placing a spatial
light modulator, such as a DMD, into the beam path of the incident light.
Imposing user-programmed masks on the incident light eliminates the need for
an arm to measure the spatial profile of the incident light, and hence is
called “computational ghost imaging.”

The ghost imaging reconstruction algorithm can be boiled down to solving a
linear matrix inversion problem:

$\mathbf{b}=\mathbf{A}\mathbf{x},$ (1)

where $\mathbf{b}$ is the bucket detector reading, $\mathbf{A}$ is the matrix
of the incident light spatial profiles, and $\mathbf{x}$ is the unknown
sample. If we denote the number of measurements by $m$ and number of pixels by
$p$, then we have $\mathbf{b}\in\mathbb{R}^{m\times 1}$,
$\mathbf{A}\in\mathbb{R}^{m\times p}$, and $\mathbf{x}\in\mathbb{R}^{p\times
1}$. In most ghost imaging problems, one can choose from a collection of
established algorithms to obtain an approximate solution to the unknown
$\mathbf{x}$ in a linear system of the form of Eq. 1.

Because the total emitted charge from a photocathode is proportional to the
product of the drive laser spatial profile and the QE map, measuring the QE
fits the scheme of classical ghost imaging. Here, the bucket detector reading
is the total charge emission from the cathode, and the sample to be imaged is
the QE map. By relating the set of varying drive laser spatial profiles
($\mathbf{A}$) to its respective total charge emission measurements
($\mathbf{b}$), it is possible to retrieve the spatial features of the QE map
($\mathbf{x}$). A practical implementation of QE ghost imaging requires only
synchronized measurements of the drive laser profile and the emitted charge,
which are already available at most accelerator facilities.

For Eq. 1 to be solvable, the rows of $\mathbf{A}$ should be independent, i.e.
the laser profile should change from shot-to-shot. In computational ghost
imaging, the variation is provided by intentionally controlling the laser
profile, e.g. with a DMD [17]. However, ghost imaging can also exploit natural
variation in the drive laser profile to measure a QE map without interrupting
normal accelerator operation. Even a small degree of variation, if
uncorrelated, can be utilized by acquiring a larger number of examples. As
data acquisition is completely passive, the possibility exists to collect
large data sets parasitically over an extended period of time, capturing more
variation from the natural spatial “jitter” of the drive laser.

##  III Experimental Demonstration

Though our goal is to exploit the random variation of the cathode laser, we
start by intentionally varying the laser illumination to evaluate the limits
of the reconstruction and the required variation in the illumination pattern.

At the UCLA Pegasus beamline, we insert a DMD into the laser path to control
digitally the drive laser profile. The use of dynamical and digitally
controllable laser shaping techniques for the drive laser has been
demonstrated in photoinjectors, as in Refs. [18, 19, 10]. In our case, the DMD
enables quantitative comparison between the traditional scanning method and
our proposed ghost imaging method. First, we obtain a ground truth QE using a
raster scan. Second, we generate a sequence of random masks on the drive laser
profile. Finally, we use a mirror controlled by a piezoelectric motor located
after the DMD to shift the position of the illumination pattern on the
cathode, emulating the transverse position jitter in the laser. Artificially
increasing the amount of jitter enables a test of the technique using fewer
measurements. Post-experiment, we analyze the laser variation for each data
set, and compare QE map reconstructions between the traditional raster scan
and the two variations that utilize the ghost imaging method. We perform
simulations using the experimental illumination patterns to guide the choice
of the reconstruction algorithm hyper-parameters, as well as analyze the
ability of random mask and laser jitter scans to retrieve QE maps.

In the following section, we demonstrate a practical experimental
implementation of the technique at the linac coherent light source (LCLS)
where data is instead acquired in parasitic mode, using the intrinsic jitter
of the drive laser profile to reconstruct the QE map of the photocathode.

###  III.1 Pegasus: Experimental Setup

Figure 1: Simplified schematic of the UCLA experiment. A 266 nm laser
illuminates 240 $\times$ 240 pixels on the DMD, which contains a programmable
mask. The transversely-shaped laser then reflects off of a UV-enhanced metal
mirror that is mounted on a piezoelectric kinematic mirror mount that can be
electronically adjusted along two axes. The laser is then imaged onto a Cu
cathode with a magnification factor of 3.8 using a lens with a 75 cm focal
length. A small portion of the beam is split using a beam-splitter upstream of
the cathode and is directed onto a screen located at the imaging plane of the
lens and monitored by a virtual cathode camera (VCC).

The experiments at the UCLA Pegasus beam line [20] use a 100 fs rms 266 nm
laser pulse to strike a copper cathode in the high field of an S-band 1.6 cell
radiofrequency (RF) gun to generate the 3.21 MeV kinetic energy electron beam.
The transverse profile of the injector laser is controlled using a Texas
Instruments’ DLP-7000 DMD with a modified window to allow ultraviolet (UV)
transmission, as discussed in [19]. A 3 mm full width laser spot fully
illuminates 240 x 240 pixels on the device. Due to the grating-like structure
of the DMD, the shaped laser is dispersed into several diffraction orders and
a pulse-front tilt (PFT) is introduced by the device. We use a 1800 lines/mm
diffraction grating upstream of the DMD to compensate for this PFT. The
brightest central order reflected from the DMD is selected and imaged onto the
cathode with a magnification factor of 3.8 using a 75 cm focal length lens. A
beam splitter upstream of the cathode directs a small portion of the laser to
a screen placed at the imaging plane and is monitored by a standard CCD
referred to as the virtual cathode camera (VCC). Because of the significant
losses on the grating, DMD, and transport line, less than 10% of the input UV
energy reaches the cathode. The input UV laser is also limited to
$<20~{}\upmu$J to avoid DMD damage.

After the RF gun, the electron beam is focused by a solenoid before a
fluorescent screen that is imaged with a standard CCD camera. The electron
beam and photoinjector laser profile images are collected synchronously using
an external trigger and labeled using a digital timestamp. In post analysis,
all background from the electron beam image is removed and a single-pixel
“bucket sum” is obtained by integrating the signal. This bucket sum is
directly proportional to the emitted charge.

The ground truth QE map is obtained using a raster scan data set. A single 16
$\times$ 16 DMD macropixel, defined as squares with widths of 16 pixels, was
turned on for each scanning point. The corresponding emitted charge is
obtained by integrating the electron image after background subtraction.
Similarly, the total laser power for each beam shot is measured by integrating
the intensity on the VCC image. The ground truth QE is mapped out by filling
in the ratio of charge emission divided by the total laser power for each
scanning data point on the VCC screen, as shown in Fig. 4 (left). The
resolution associated with the size of the macropixel at the cathode plane
corresponds to $57\pm 5~{}\upmu$m, which is comparable to the resolution
obtained in traditional scanning methods and is sufficient to capture the QE
features on the photocathode used for this experiment.

To test the ghost imaging technique, two main data sets were collected during
the experiment. First, for the “random” data set, we displayed 350 random
(binary) masks on the DMD. In each random mask, each of the 16 $\times$ 16
macropixels in the illuminated 240 $\times$ 240 pixel region was turned on
with 50% probability. Second, the “jitter” data set consisted of 361
measurements with all pixels on the DMD turned on. We varied the injector
laser profile on the cathode by remotely steering the beam using a
piezoelectric motor controlled mirror. The beam was steered by a maximum of
0.2 mm (nearly 25 $\%$ of the laser spot size) horizontally and vertically on
the cathode from the starting reference position. Although this variation is
larger than that observed in typical photoinjector setups, the number of
measurements is orders of magnitude smaller than the data that can be
collected passively at facilities such as LCLS.

For both data sets, the corresponding VCC images are cropped to maximally
encompass the laser beam and then subsequently downsized to a 30$\times$30
grid. We downsize the VCC images to reduce the number of free parameters for
our regression algorithm and avoid an underdetermined regression problem that
makes solving Eq. 1 more difficult. Downsizing can be avoided at the cost of
taking more measurements, use of stronger regularization, and longer
computational reconstruction times.

###  III.2 Pegasus: Data Analysis and Simulations

In order to quantify the variation of the drive laser for a particular data
set we consider two methods of analysis. The first method examines the
coefficient of variation (CV), also known as the relative standard deviation,
for each pixel across the VCC data set. The second method uses principal
component analysis (PCA), a statistical procedure where the variables of a
data set are transformed into an ordered orthogonal basis [21]. The first
principal component (PC) correponds to an eigenvector of the data set
containing the largest variance. PCA allows us to capture information
concerning the linear independence of laser profiles across different
measurements. If there are enough eigenvectors accounting for a majority of
the total drive laser spatial variation, then there is a significant amount of
laser variation that ensures shot-to-shot measurements are sufficiently
independent.

Figure 2: Drive laser variation analysis for UCLA jitter scans (top row) and
random scans (bottom row). The left-column shows the CV for each pixel. Note
that edge pixels that have a mean of approximately zero have been discarded to
avoid division by zero errors. The right-column shows the cumulative
percentage of total variation contained within the eigenvectors of the data
set. The dashed lines indicates the number of eigenvectors containing 80% of
the total variation. This was 5 eigenvectors for the jitter scans and 67
eigenvectors for the random scans. The total number of eigenvectors for both
data sets was 900.

The analysis for the UCLA random and jitter scans can be seen in Fig. 2. As
expected, the CV analysis shows that the random scan contains more laser
variation across the region of interest whereas the jitter scan contains
variation concentrated along the edges. We note that the checkerboard pattern
in the random scan CV figure is an artifact of DMD damage. Moreover, the PCA
results indicate that it takes more eigenvectors in the random scan to
describe the same level of cumulative percentage of total variation than the
jitter scan.

To clarify the effects of this difference in the illumination patterns, we
conduct simulations using the UCLA jitter and random scan VCC data and
multiple user-generated QE maps each containing one 2D Gaussian-shaped hot
spot with a centroid placed at a unique location. The simulation method is
discussed in Appendix A. The results of the simulation can be seen in Figure
3. A score map is used to visualize the performance of reconstructions for hot
spots centered at a coordinate within the map. The score is defined to be:

$S=\log{\frac{1}{MSE}}$ (2)

where $S$ is the score and $MSE$ is the mean-square error between the target
QE map and the reconstruction upsized to have the same dimensions as the
target. With this definition, a higher score implies a more accurate
reconstruction. The reconstructions for both the jitter and random scans
appear to be well-behaved for QE hot spots centered in any arbitrary location.
As expected the random mask data set provides more accurate reconstructions as
it contains more spatial variation in the illumination series (see Fig. 2).
The jitter data set results in non-physical artifacts, but still recovers the
cathode hot spot in each case. As we compare different experiments, it is
important to note that the amount of variation required also greatly depends
on the number of samples as well as the SNR in the bucket sum acquisition.

Figure 3: Simulation results using UCLA jitter (top) and random scan (bottom)
data sets for a collection of QE maps each containing one 2D Gaussian-shaped
hot spot with a centroid placed at a unique location. The left-most image
shows a subset of the generated QE maps. The score maps (middle) represent the
performance of a reconstruction with a target hot spot centered at the
coordinate within the map. The score is defined in Eq. 2. The right-most
figures show the best and worst scored reconstructions side-by-side with the
target QE map. Note that the reconstructions have the same size of as the VCC
images, and are upsized to match the dimensions of the target QE map during
score calculations. Some of the jitter reconstructions includee artifacts, but
both methods identify the QE hotspot.

Finally, we reconstruct the QE map using the alternating direction method of
multipliers (ADMM) algorithm [22]. The hyperparameters for ADMM are selected
to optimize a simulated QE map consisting of a single Gaussian-shaped hot spot
centered at the origin. The choice to use this map for the simulation comes
from the fact that many photoinjector based accelerator facilities utilize a
photocathode with a QE that contains a single hot spot (see for example [23,
24, 25]). The reconstructions are presented in Fig. 4.

Figure 4: QE map reconstructions. The top row shows example laser profiles
used to generate the QE maps seen on the bottom row. Left: Ground truth QE map
obtained by raster scan. Middle: reconstruction using the random mask data
set. Right: reconstruction using the jitter data set. The random and jitter
data sets successfully reconstruct the hot spot seen in the upper right corner
of the ground truth.

The quality of the reconstructions is quantified by the MSE with the raster
scan ground truth. Both the random and jitter data sets capture the major
feature of a hot spot on the upper right corner, consistent with the ground
truth. Moreover, the random data set captures low-level QE variations below
the hot spot that are not captured by the raster scan. This is also partially
captured by the jitter scan. This feature may be real and only resolvable by
the random and jitter scans due to Fellgett’s advantage.

In sum, we are able to reconstruct the quantum efficiency of the cathode using
either the DMD to create random patterns or passively using the jitter from
the laser pointing. Using simulation, we show how the CV and PCA can help
assess the amount of laser spatial variation needed for a reconstruction. As
expected, the jitter data set performs worse than the random set, but is still
capable of recovering major features in the QE map. In the next section we
will show how the method translates to operation at LCLS, where the natural
spatial jitter in the illumination is smaller, but it is possible to record
orders of magnitude more data.

###  III.3 LCLS: Parasitic Demonstration

Figure 5: Examples of single shot LCLS injector laser profiles over the course
of 10 days. Camera intensity is in arbitrary units.

At LCLS, the injector laser consists of a Ti:Sapphire laser system, producing
2 ps laser pulses at 760 nm wavelength. The infrared laser is then tripled to
ultraviolet wavelength (253 nm) and generates electrons by striking a copper
photocathode [26]. In normal operation, the laser spot size has a diameter of
$\sim$1 mm on the photocathode, emitting electron bunches of $\sim$250 pC.
Similarly to the UCLA Pegasus setup, before the injector laser reaches the
cathode, it goes through a beam splitter where a small portion of the beam is
imaged onto a VCC which captures the injector laser profile on the
photocathode. We record the injector laser profile synchronously with the
corresponding charge measured by a beam position monitor. Over the course of
ten days, we took data parasitically for about two hours each day during both
normal operation for XFEL and accelerator beam studies. After filtering out
shots where the charge drops below 100 pC or when the laser is shuttered, we
obtain on average 6949 shots per day. We crop the VCC images to encompass the
full laser beam and downsize the images to a 20$\times$20 grid to reduce the
number of free parameters. In the downsized image, each pixel size corresponds
to $\sim$76 $\upmu$m, which is less than 10% of the beam diameter and
therefore retains the features in the laser beam.

One challenge for collecting data parasitically is the charge-feedback system,
which continuously tunes the injector laser power to maintain an average
charge level around 250 pC. Because we do not have measurements of the
underlying cause, we account for the charge feedback by normalizing the VCC
images by the integrated laser intensity of a running average over 100 shots.
Note that the normalization implicitly assumes that the drift in charge is not
caused by changes to the QE map (which we assume to be fixed during the
measurement) or the laser profile jitter (which is fast compared to the
feedback). We have also verified that choosing to average over 100 shots does
not significantly change the PCA curve as shown in Fig. 6, and, therefore,
does not remove the shot-to-shot fluctuations in the laser profiles.

Figure 6: Cumulative variation explained by number of eigenvectors from PCA of
the LCLS laser profiles. To capture 80% of the total variation, it takes 1 to
5 eigenvectors depending on the day. The total number of eigenvectors is 400.

Figure 5 shows the natural spatial jitter apparent in the LCLS injector laser
on four of the ten days. In order to assess the variation of the laser
profiles, we apply the PCA decomposition method described in the Sec. III.2 to
the LCLS data. Figure 6 shows the PCA results of the LCLS data by day. It
takes similar number of eigenvectors to cover 80% of the total cumulative
variation compared to the UCLA jitter scan, and significantly fewer
eigenvectors compared to the UCLA random scan. This indicates that the natural
jitter provides comparable variation as inducing jitter by steering mirrors
and less variation than introducing random binary patterns. As shown later,
this amount of variation is sufficient to produce a consistent QE
reconstruction over days. The PCA metric provides qualitatively the level of
validity of the reconstructions using the data from different days.
Furthermore, from the steering mirror movement at LCLS we see there was large
movement of the mirror between day 6 and 7 due to the accelerator’s
operational interruptions, which is known to cause shifts due to backlash in
the steering motors. Although there has been no intentional movement of the
laser position on the cathode, we estimate from the drift in the mirror motor
movement (see Fig. 10) that the beam moved by approximately 10% of the beam
size.

Next, we conduct a simulation study with a user-generated QE map similar to
that described in Appendix A. We simulate the charge measurement by Eq. 3
using the measured VCC images and a Gaussian noise with an

Figure 7: Simulated ground truth (at left) and day-by-day reconstruction using
recorded LCLS VCC images and simulated bucket measurements. A faint circular
ring of high QE appears consistently on the aperture of the laser profile.
This is in error with the simulated ground truth, implying that it is an
artifact of the reconstruction algorithm.

Figure 8: Reconstruction of LCLS data by day. Reconstructions are relative to
the center of the laser beam, so a shift in beam centroid on the cathode will
correspond to a shift in the reconstructed cathode QE. There is a consistent
hotspot just below and to the left of the laser centroid. The strong circular
ring of high QE at the edge of the laser aperture may be an artifact from the
reconstruction algorithm.

SNR of 71. We scan the hyperparameters to optimize the MSE of the
reconstruction, and the results are presented in Fig. 7.

Using the same hyperparameters that optimize the reconstruction for the
simulated QE, we feed the day-by-day data into the ADMM algorithm and obtain
the reconstructed QE maps shown in Fig. 8. The results show a consistent hot
spot at the lower part of the beam region. From the steering mirror movement
shown in Fig. 10, between days 6 and 7, the laser moves left and up, which
explains why the hotspot of the cathode QE moves right and down in the
reconstructions over the ten days (Fig. 8). We note that a circular ring at
the upper edge of the laser aperture appears in both simulation and
reconstruction. Therefore, it may be an artifact of the reconstruction
algorithm rather than a real feature.

Without a dedicated measurement shift, we have no way to measure the ground
truth QE for comparison with Fig. 8. However, by reconstructing a consistent
QE map repeatedly over many days, we have some degree of cross-validation of
the result. We also highlight that the reconstructed QE does not appear to be
related to the

average laser profile; the laser shape changes significantly during the data
collection (Fig. 5) without changing the QE reconstructions in Fig. 8.

Finally, we emphasize that these reconstructions have used a relatively small
number of measurements. Given the passive nature of the method, extensive data
sets can be collected without interfering with operation. For example, at
LCLS-II it will be feasible to measure millions of shots per days, enabling
reconstructions with relatively small amounts of jitter in the laser.

##  IV Conclusions

In conclusion, we present a novel method for measuring the spatial features in
QE of injector cathodes based on a ghost imaging reconstruction framework. The
method is validated on ad-hoc measurements at the UCLA Pegasus photoinjector,
showing equivalent results to a common raster scan. This experiment is used to
qualitatively assess the variation in the illumination pattern needed to
obtain a reliable reconstruction. The method is then applied to passive
measurements taken during routine operation of LCLS to show the main advantage
of being able to run parasitically off the normal operation.

Cathode QE ghost imaging holds promise for electron-source based user
facilities where dedicated time to study the performances of the injector is
limited, and the spatial variation of the QE is a critical quantity affecting
the final beam brightness and machine performance. More generally, we believe
ghost imaging can be a valuable tool at accelerators, where noisy probes are
common and opportunities for dedicated studies are limited [15, 27]. The ghost
imaging philosophy of “measurement is easier than control” can find
applications for a wide range of accelerator problems.

###### Acknowledgements.

We would like to thank Brendan O’Shea for inspiration to study ghost imaging
for QE maps, and Lauren Alsberg, Franz-Josef Decker, Dave Dowell, and Tim
Maxwell for helpful discussions. This work was partially supported by STROBE
National Science Foundation Science & Technology Center, Grant No.
DMR-1548924.

##  Appendix A Simulation

Figure 9: Overview of simulation using UCLA experimental data. An example of
an experimental VCC image (top-right), a user-generated QE map (top-left) and
simulated electron beam (top-middle) are shown. Bucket sums are calculated
integrating all pixels of the simulated e-beam and and adding random Gaussian
noise $\epsilon_{i}$, as described by Eq. 3. Note that the VCC image is
upsized to match the size of the QE map. The bottom-middle plot shows example
simulated bucket sums for all random scan measurements. The blue line shows
the ‘noiseless’ bucket sums and the red dashed line shows boundaries of $\pm
5\epsilon$ within which the simulated bucket sums are contained. An example
reconstruction using an experimental SNR of 7.8 and $N=$ 350 random masks is
given at the bottom. The reconstruction is the same size as the original VCC
images.

Figure 10: Steering mirror movement across the days of data acquisition. The
data between day 6 and 7 are discarded due to operational interruptions. Top
panel: horizontal movement. Bottom panel: vertical movement. The positive sign
in horizontal direction means the laser moves right, in vertical up.

To select ADMM hyperparameters for the experimental reconstructions, as well
as characterize the spatial variation of the measured laser profiles needed to
resolve arbitrary QE maps, we conduct measurement simulations using the
experimental VCC images and user-generated QE maps. An example simulation
using UCLA data is described in Fig. 9. The bucket sum for the
$i^{\mathrm{th}}$ example is calculated as

$b_{i}=\mathbf{a}_{i}\mathbf{x}+\varepsilon_{i}$ (3)

where $\mathbf{a}_{i}$ is the $i^{\mathrm{th}}$ row of $\mathbf{A}$, and
$\varepsilon_{i}=\mathcal{N}(0,\varepsilon)$ is zero-centered Gaussian noise
with standard deviation $\varepsilon$. The noise level, $\varepsilon$, is the
bucket mean divided by the experimental SNR. As several bucket sums were
obtained for a single mask during experiment, the experimental SNR was
calculated as follows:

$SNR=\frac{1}{N}\sum_{i=1}^{N}\frac{\mu_{i}}{\sigma_{i}}$ (4)

where $N$ is the total number of unique masks obtained during the experiment,
and $\mu_{i}$ and $\sigma_{i}$ are the mean and standard deviation
respectively of the bucket sums for a single mask.

ADMM hyperparameters for reconstructions are selected by scanning through a
range of values. Hyperparameters that correspond to the lowest MSE for
simulated reconstructions are separately selected for different datasets.

##  Appendix B LCLS mirror movement

The movement of the steering mirror upstream of the VCC is recorded during the
days of data acquisition (Fig. 10). The data between days 6 and 7 are
discarded due to operational interruptions.

## References

  * Fraser  _et al._ [1986] J. Fraser, R. Sheffield, and E. Gray, A new high-brightness electron injector for free electron lasers driven by rf linacs, Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 250, 71 (1986).
  * Serafini and Rosenzweig [1997] L. Serafini and J. B. Rosenzweig, Envelope analysis of intense relativistic quasilaminar beams in rf photoinjectors: ma theory of emittance compensation, Physical Review E 55, 7565 (1997).
  * Dowell [2016] D. H. Dowell, Sources of emittance in rf photocathode injectors: Intrinsic emittance, space charge forces due to non-uniformities, rf and solenoid effects, arXiv preprint arXiv:1610.01242 (2016).
  * Zhou  _et al._ [2002] F. Zhou, I. Ben-Zvi, M. Babzien, X. Y. Chang, A. Doyuran, R. Malone, X. J. Wang, and V. Yakimenko, Experimental characterization of emittance growth induced by the nonuniform transverse laser distribution in a photoinjector, Phys. Rev. ST Accel. Beams 5, 094203 (2002).
  * Zhou  _et al._ [2012a] F. Zhou, A. Brachmann, P. Emma, S. Gilevich, and Z. Huang, Impact of the spatial laser distribution on photocathode gun operation, Phys. Rev. ST Accel. Beams 15, 090701 (2012a).
  * Riddick  _et al._ [2013] B. Riddick, E. Montgomery, R. Fiorito, H. Zhang, A. Shkvarunets, Z. Pan, and S. Khan, Photocathode quantum efficiency mapping at high resolution using a digital micromirror device, Physical Review Special Topics-Accelerators and Beams 16, 062802 (2013).
  * Pelliccia  _et al._ [2016] D. Pelliccia, A. Rack, M. Scheel, V. Cantelli, and D. M. Paganin, Experimental x-ray ghost imaging, Physical review letters 117, 113902 (2016).
  * Yu  _et al._ [2016] H. Yu, R. Lu, S. Han, H. Xie, G. Du, T. Xiao, and D. Zhu, Fourier-transform ghost imaging with hard x rays, Physical review letters 117, 113901 (2016).
  * Khakimov  _et al._ [2016] R. I. Khakimov, B. Henson, D. Shin, S. Hodgman, R. Dall, K. Baldwin, and A. Truscott, Ghost imaging with atoms, Nature 540, 100 (2016).
  * Li  _et al._ [2018] S. Li, F. Cropp, K. Kabra, T. Lane, G. Wetzstein, P. Musumeci, and D. Ratner, Electron ghost imaging, Physical review letters 121, 114801 (2018).
  * Fellgett [1949] P. B. Fellgett, On the ultimate sensitivity and practical performance of radiation detectors, J. Opt. Soc. Am. 39, 970 (1949).
  * Candès and Wakin [2008] E. J. Candès and M. B. Wakin, An introduction to compressive sampling [a sensing/sampling paradigm that goes against the common knowledge in data acquisition], IEEE signal processing magazine 25, 21 (2008).
  * Katz  _et al._ [2009] O. Katz, Y. Bromberg, and Y. Silberberg, Compressive ghost imaging, Applied Physics Letters 95, 131110 (2009), https://doi.org/10.1063/1.3238296 .
  * Tian  _et al._ [2014] L. Tian, X. Li, K. Ramchandran, and L. Waller, Multiplexed coded illumination for fourier ptychography with an led array microscope, Biomed. Opt. Express 5, 2376 (2014).
  * Lane and Ratner [2019] T. J. Lane and D. Ratner, What are the advantages of ghost imaging? multiplexing for x-ray and electron imaging, arXiv preprint arXiv:1907.12178 (2019).
  * Shapiro [2008] J. H. Shapiro, Computational ghost imaging, Phys. Rev. A 78, 061802 (2008).
  * Caravaca-Aguirre  _et al._ [2019] A. M. Caravaca-Aguirre, S. Singh, S. Labouesse, M. V. Baratta, R. Piestun, and E. Bossy, Hybrid photoacoustic-fluorescence microendoscopy through a multimode fiber using speckle illumination, APL Photonics 4, 096103 (2019), https://doi.org/10.1063/1.5113476 .
  * Maxson  _et al._ [2015] J. Maxson, H. Lee, A. C. Bartnik, J. Kiefer, and I. Bazarov, Adaptive electron beam shaping using a photoemission gun and spatial light modulator, Physical Review Special Topics-Accelerators and Beams 18, 023401 (2015).
  * Li  _et al._ [2017] S. Li, S. Alverson, D. Bohler, A. Egger, A. Fry, S. Gilevich, Z. Huang, A. Miahnahri, D. Ratner, J. Robinson, _et al._ , Ultraviolet laser transverse profile shaping for improving x-ray free electron laser performance, Physical Review Accelerators and Beams 20, 080704 (2017).
  * Alesini  _et al._ [2015] D. Alesini, A. Battisti, M. Ferrario, L. Foggetta, V. Lollo, L. Ficcadenti, V. Pettinacci, S. Custodio, E. Pirez, P. Musumeci, _et al._ , New technology based on clamping for high gradient radio frequency photogun, Physical Review Special Topics-Accelerators and Beams 18, 092001 (2015).
  * Jolliffe [2002] I. T. Jolliffe,  _Principal component analysis_ , 2nd ed., Spring Series in Statistics (Springer New York, 2002).
  * Boyd  _et al._ [2011] S. Boyd, N. Parikh, E. Chu, B. Peleato, J. Eckstein, _et al._ , Distributed optimization and statistical learning via the alternating direction method of multipliers, Foundations and Trends® in Machine learning 3, 1 (2011).
  * Cultrera  _et al._ [2011] L. Cultrera, J. Maxson, I. Bazarov, S. Belomestnykh, J. Dobbins, B. Dunham, S. Karkare, R. Kaplan, V. Kostroun, Y. Li, X. Liu, F. Löhl, K. Smolenski, Z. Zhao, D. Rice, P. Quigley, M. Tigner, V. Veshcherevich, K. Finkelstein, D. Dale, and B. Pichler, Photocathode behavior during high current running in the cornell energy recovery linac photoinjector, Phys. Rev. ST Accel. Beams 14, 120101 (2011).
  * Barday  _et al._ [2013] R. Barday, A. Burrill, A. Jankowiak, T. Kamps, J. Knobloch, O. Kugeler, A. Matveenko, A. Neumann, M. Schmeißer, J. Völker, P. Kneisel, R. Nietubyc, S. Schubert, J. Smedley, J. Sekutowicz, and I. Will, Characterization of a superconducting pb photocathode in a superconducting rf photoinjector cavity, Phys. Rev. ST Accel. Beams 16, 123402 (2013).
  * Stephan  _et al._ [2010] F. Stephan, C. H. Boulware, M. Krasilnikov, J. Bähr, G. Asova, A. Donat, U. Gensch, H. J. Grabosch, M. Hänel, L. Hakobyan, H. Henschel, Y. Ivanisenko, L. Jachmann, S. Khodyachykh, M. Khojoyan, W. Köhler, S. Korepanov, G. Koss, A. Kretzschmann, H. Leich, H. Lüdecke, A. Meissner, A. Oppelt, B. Petrosyan, M. Pohl, S. Riemann, S. Rimjaem, M. Sachwitz, B. Schöneich, T. Scholz, H. Schulze, J. Schultze, U. Schwendicke, A. Shapovalov, R. Spesyvtsev, L. Staykov, F. Tonisch, T. Walter, S. Weisse, R. Wenndorff, M. Winde, L. v. Vu, H. Dürr, T. Kamps, D. Richter, M. Sperling, R. Ovsyannikov, A. Vollmer, J. Knobloch, E. Jaeschke, J. Boster, R. Brinkmann, S. Choroba, K. Flechsenhar, K. Flöttmann, W. Gerdau, V. Katalev, W. Koprek, S. Lederer, C. Martens, P. Pucyk, S. Schreiber, S. Simrock, E. Vogel, V. Vogel, K. Rosbach, I. Bonev, I. Tsakov, P. Michelato, L. Monaco, C. Pagani, D. Sertore, T. Garvey, I. Will, I. Templin, W. Sandner, W. Ackermann, E. Arévalo, E. Gjonaj, W. F. O. Müller, S. Schnepp, T. Weiland, F. Wolfheimer, J. Rönsch, and J. Rossbach, Detailed characterization of electron sources yielding first demonstration of european x-ray free-electron laser beam quality, Phys. Rev. ST Accel. Beams 13, 020704 (2010).
  * Akre  _et al._ [2008] R. Akre, D. Dowell, P. Emma, J. Frisch, S. Gilevich, G. Hays, P. Hering, R. Iverson, C. Limborg-Deprey, H. Loos, _et al._ , Commissioning the linac coherent light source injector, Physical Review Special Topics-Accelerators and Beams 11, 030703 (2008).
  * Ratner  _et al._ [2019] D. Ratner, J. Cryan, T. Lane, S. Li, and G. Stupakov, Pump-probe ghost imaging with sase fels, Phys. Rev. X 9, 011045 (2019).
  * Zhou  _et al._ [2012b] F. Zhou, A. Brachmann, F. Decker, P. Emma, S. Gilevich, R. Iverson, P. Stefan, and J. Turner, High-brightness electron beam evolution following laser-based cleaning of a photocathode, Physical Review Special Topics-Accelerators and Beams 15, 090703 (2012b).