# Algorithmic Bias, Generalist Models,
and Clinical Medicine

Geoff Keeling  Google Research
gkeeling [at] google.com

(May 2023)

###### Abstract

The technical landscape of clinical machine learning is shifting in ways that
destabilize pervasive assumptions about the nature and causes of algorithmic
bias. On one hand, the dominant paradigm in clinical machine learning is
narrow in the sense that models are trained on biomedical datasets for
particular clinical tasks such as diagnosis and treatment recommendation. On
the other hand, the emerging paradigm is generalist in the sense that general-
purpose language models such as Google’s BERT and PaLM are increasingly being
adapted for clinical use cases via prompting or fine-tuning on biomedical
datasets. Many of these next-generation models provide substantial performance
gains over prior clinical models, but at the same time introduce novel kinds
of algorithmic bias and complicate the explanatory relationship between
algorithmic biases and biases in training data. This paper articulates how and
in what respects biases in generalist models differ from biases in prior
clinical models, and draws out practical recommendations for algorithmic bias
mitigation.

Keywords: Large Language Models, Machine Learning, Artificial Intelligence,
BERT, Pre-Trained Model, Medicine, Healthcare, Medical AI, Bias, Fairness,
Ethics

##  1 Introduction

Statistical models, including deep neural networks trained via machine
learning, are used in medicine for risk assessment, diagnosis, and treatment
recommendation, among other clinical use cases.111For an introduction to
machine learning in medicine see Rajkomar et al. (2019). These models can
exhibit biases. For example, a model that classifies dermatologic conditions
based on images of skin lesions may be less accurate for images featuring
darker skin tones compared to lighter skin tones (Groh et al., 2021, Daneshjou
et al., 2022). Typically, such biases arise because, mutatis mutandis, darker
skin tones are underrepresented in the model’s training data such that the
model either fails to register predictively salient statistical patterns that
hold for the underrepresented group, or else generalizes a pattern that holds
for members of the group sampled but does not hold for that group at the
population level.222Performance biases can also arise, among other reasons,
from misrepresentative training data such as datasets that employ proxy
variables or data labels that systematically distort the circumstances of a
disadvantaged group (Obermeyer et al., 2019, Jiang and Nachum, 2020). See
Section 2.3 for discussion. These biases are morally significant because at
scale biased models threaten to sustain or exacerbate existing inequities in
healthcare (Panch et al., 2019, Grote and Keeling, 2022a).

What is central to the conception of algorithmic bias above is the ontic
assumption that algorithmic biases are, roughly, measurable performance
disparities across demographic groups; and the explanatory assumption that
such disparities admit explanation in terms of data biases like
underrepresentation or misrepresentation of groups in training data. These
assumptions are pervasive. Take the US Food and Drug Administration’s (FDA)
2021 report on Software as a Medical Device (SaMD) systems that use machine
learning: ‘[b]ecause [such] systems are […] trained using data from historical
datasets, they are vulnerable to bias - and prone to mirroring biases present
in the data’ (Food and Drug Administration, 2021a, p.5). This picture is
echoed in the FDA’s 2021 principles on ‘Good Machine Learning Practice for
Medical Device Development,’ released alongside the UK Medicines and
Healthcare Products Regulatory Agency and Health Canada. According to these
principles, developers should ensure that ‘datasets are representative of the
target population [to] manage any bias [and] promote appropriate and
generalizable performance across the intended population’ (Food and Drug
Administration, 2021b, p.2). Likewise, in the scientific literature, Norori et
al. (2021, p.1) state that certain subpopulations ‘[are] absent or
misrepresented in existing biomedical datasets [such that clinical models are]
prone to reinforcing bias’ (c.f. Obermeyer et al., 2019, Challen et al., 2019,
Cirillo et al., 2020).

This paper argues that the technical landscape of clinical machine learning is
shifting in ways that destabilize and reframe these pervasive assumptions
about the nature and causes of algorithmic bias. On the one hand, the dominant
paradigm in clinical machine learning is narrow in the sense that models are
trained on biomedical datasets for particular clinical tasks such as diagnosis
and treatment recommendation. On the other hand, the emerging paradigm is
generalist in the sense that general-purpose language models such as Google’s
BERT (Devlin et al., 2018) and Meta’s OPT (Zhang et al., 2022) are
increasingly being adapted for clinical use cases via fine-tuning on
biomedical datasets (Bommasani et al., 2021, pp.54-58). Indeed, BERT has
already been adapted for a range of clinical functions including BioBERT for
biomedical text mining (Lee et al., 2020), G-BERT for treatment recommendation
(Shang et al., 2019), Med-BERT for disease prediction (Rasmy et al., 2021),
and ClinicalBERT for predicting hospital readmission (Huang et al., 2019).
Many of these next-generation models provide substantial performance gains
over prior clinical models, but at the same time introduce novel kinds of
algorithmic bias, and complicate the explanatory relationship between
demographic performance disparities and biases in training data. This paper
articulates how and in what respects biases in generalist models differ from
biases in previous clinical models, and draws out practical recommendations
for algorithmic bias mitigation in SaMD technologies built using generalist
language models.

The paper proceeds as follows. Section 2 introduces the narrow paradigm
alongside the ontic and explanatory assumptions about algorithmic bias that it
gave rise to. Section 3 characterizes the emerging generalist paradigm and
shows how biases in this paradigm challenge the prevailing assumptions.
Section 4 explores bias mitigation strategies for generalist models. Section 5
concludes.

##  2 The Narrow Paradigm

###  2.1 Narrow Models and Supervised Learning

The core idea of the narrow paradigm is that machine learning models are
trained with a particular clinical task in mind. An example of a clinical task
is predicting whether a nodule depicted in a computed tomography (CT) image is
malignant or benign (Uthoff et al., 2019). Another is predicting based on
features of a patient’s medical history whether or not that patient will be
readmitted to hospital within 30 days if discharged (Hasan et al., 2010).

The dominant approach to machine learning within the narrow paradigm is called
supervised learning. Here a model is trained to predict an outcome variable
based on one or more input features via a large number of examples. So given a
dataset comprising a series of CT images each depicting a nodule correctly
labeled as ‘benign’ or ‘malignant,’ a model can learn which features of the
images predict malignancy through a trial-and-error process where the model
parameters are iteratively updated to correct for erroneous predictions
(Uthoff et al., 2019). Supervised learning accounts for much of the recent
success in clinical machine learning. This includes image classification
models that rival the performance of clinicians at dermatologic diagnosis
(Tschandl et al., 2019), and similarly performant models for radiological
tasks such as image-based diagnosis, segmentation, and identification (Kelly
et al., 2022).

Sometimes supervised learning is supplemented with a technique called transfer
learning (Weiss et al., 2016). Standardly, training data reflects the kind of
inputs and outputs that the model will operate with in practice. Hence a model
for identifying malignant nodules is typically trained on labeled images of
malignant and benign nodules. But in some domains - including medicine -
training data is expensive to collate because experts are required to label
the data and the quantity of data needed is substantial (Rahimi et al., 2021).
Transfer learning offers a partial solution. Here the model is pre-trained on
a generic task and then fine-tuned for a specific task. The idea is that
statistical associations learned in the generic task are applicable in (and
thus transfer to) the domain-specific task. This reduces the amount of
clinical data that is required to train the model. Hence a model for
identifying malignant nodules might first be trained to classify images on a
generic image dataset such as ImageNet (Deng et al., 2009), and subsequently
be fine-tuned to detect malignant nodules by holding fixed the model weights
in the main body of the neural network that account for lower level features
such as edges and corners, and re-training the top layers to account for
higher level clinically significant features using the nodule image dataset
(c.f. Chowdhury et al., 2021).

Importantly: Whether or not transfer learning is used the background
assumption is that each model is trained to perform a particular clinical
task. Pervasive assumptions about the nature and causes of algorithmic bias in
medicine have developed on the backdrop of this narrow paradigm. The next two
sections articulate and make precise these assumptions.

###  2.2 What are Biases?

On first approximation the ontic assumption holds that algorithmic biases
admit analysis in terms of measurable performance disparities across
demographic groups. So one respect in which a model for identifying the
presence or absence of a particular dermatologic condition based on images of
skin lesions might be biased is if the area under the receiver operating
characteristic curve (AUC-ROC) is lower for images with Fitzpatrick skin types
I-II compared to images with Fitzpatrick skin types V-VI (Daneshjou et al.,
2022). Here AUC-ROC is a metric that measures the degree to which a binary
classifier is able to distinguish between the two classes (in this case the
presence or absence of the relevant dermatologic condition). The implication
of different AUC-ROC scores for Fitzpatrick skin types I-II and V-VI is that
the model is better able to identify the relevant condition for White patients
compared to Black patients, as race and Fitzpatrick skin type are correlated,
albeit imperfectly (Ware et al., 2020). Performance disparities across
demographic groups like the one described is typically what is at issue when
clinical models are claimed to exhibit algorithmic bias (Hellström et al.,
2020).

Whether a model exhibits performance disparities across demographic groups is
not always clear. Reasonable people can disagree about what counts as good
performance. For example, individuals with different attitudes to risk may
disagree about the comparative importance of precision and recall in a
screening task. Here precision is the fraction of actually positive cases
among those classified as positive and recall is the fraction of actually
positive cases that end up classified as positive. Risk-neutral people who are
indifferent between errors in which actually negative cases are classified as
positive and errors in which actually positive cases are classified as
negative may advocate for the $F_{1}$ score as a performance metric, i.e. the
harmonic mean of precision and recall. In contrast, risk-averse people who
prefer actually negative cases to test positive than for actually positive
cases to test negative may advocate for an $F_{\beta}$ score, which is a
weighted harmonic mean where recall has $\beta$ times the weight of precision
for some $\beta$ $>$ 0\. Hence even if algorithmic bias is understood in the
sense of measurable demographic performance disparities, there may be
disagreements about which models are biased given the lack of consensus on how
to measure performance (c.f. Dieterich et al., 2016).333The problem is
underscored by the fact that absent equal base rates across subpopulations or
perfect predictive performance binary classifiers cannot equalize precision
and false positive/negative rates (Chouldechova, 2017). See Kleinberg et al.
(2016) for an analogous result for continuous risk scores. For discussion of
the significance of the fairness impossibility theorems for healthcare see
Grote and Keeling (2022a) and Grote and Keeling (2022b).

###  2.3 How are Biases Explained?

The explanatory assumption holds that algorithmic biases are typically
explained by biases in training data. Performance disparities obtain at the
model level because the model’s training data underrepresents or misrepresents
particular demographic groups. To use the FDA’s term, algorithmic biases are
‘prone to mirroring biases present in the data’ Food and Drug Administration
(2021a, p.5, emphasis mine). The relevant sense of mirroring is that
performance disparities which disadvantage group G are explained by
underrepresentation or misrepresentation of G in the training data.444The
implication here is not that all performance biases are explained by biases in
training data, as biases can arise at every stage of the machine learning
pipeline (Rajkomar et al., 2019). Rather, the claim is that performance biases
(at least in healthcare where demographic data biases are widespread and
pervasive) in a broad class of cases arise due to biases in datasets such as
under-representative or misrepresentative training data. Such is the extent of
data biases that it makes sense for organizations like the to orient their
general bias mitigation advice around data representativeness (c.f. Food and
Drug Administration, 2021a).

The explanatory assumption allows for multiple explanatory pathways which
chart the way that biases in model performance depend on data biases. One
common explanatory pathway that arises in cases of underrepresentative
training data is where the model generalizes a statistical association that
holds for an overrepresented group to the entire population such that the
association fails to hold for the underrepresented group. For example, HbA1c
level and stroke are associated in that poor glycemic control is a risk factor
for stroke (Mitsios et al., 2018). Yet the association between HbA1c level and
stroke for women is more pronounced for women over age 55 compared to women
under 55 (Zhao et al., 2014). Hence training data for an HbA1c based stroke
risk assessment model that undersamples women over 55 may lead the model to
generalize the association that holds for women under 55 to all women,
resulting in suboptimal predictive performance for women over 55. Here the
underrepresentation of a given demographic group in training data leads the
model to underfit for that group.

Misrepresentative training data can also lead to performance disparities. What
is at issue in some instances of misrepresentation is the use of proxy
variables that misrepresent a group’s circumstances. For example, Obermeyer et
al. (2019) found that a model used in the US to enroll sufficiently sick
patients into a narrow care program exhibited racial bias. The model was such
that, for any given risk score, Black patients at that risk score were sicker
than White patients at that risk score, where sickness is measured by number
of active chronic conditions. Patients are automatically referred into the
narrow care program only if their risk score is above some threshold. Hence
Black patients need to be sicker than White patients to qualify for automatic
enrolment in the program. The bias arose because the variable health cost was
used as a proxy for health need. Because Black and White patients matched on
health need are on average such that less money is spent on the Black patient,
the effect of using health cost as a proxy for health need was to misrepresent
the healthcare needs of the Black patients.

Other cases of misrepresentation involve misrepresentative data labels. This
can occur if there are diagnostic disparities across groups. For example, in
the United States, clinical screening tools for dementia are known to be less
reliable for racial minorities (Stephenson, 2001, Gianattasio et al., 2019).
Models trained on labeled data such that the labels reflect patterns of
systematic misdiagnosis are liable to replicate those patterns in their
predictions (Rajkomar et al., 2018, Hellström et al., 2020). Yet more
instances of misrepresentation arise given differential missingness across
groups. For example, longitudinal EHR datasets are liable to systematically
exclude vulnerable populations such as immigrants who are more likely to
receive fractured care across multiple healthcare institutions (Gianfrancesco
et al., 2018). Hence models trained on longitudinal EHR data may fail to
detect clinically significant statistical patterns that pertain to the
relevant populations. The downstream impact may then be underperformance for
certain vulnerable populations.

The significance of the explanatory assumption relates to bias mitigation. A
recent paper by Willemink et al. (2020, p.9) on ‘Preparing Medical Imaging
Data for Machine Learning’ states that ‘[i]f an AI algorithm is trained with
images from a European institution and the algorithm is used in an Asian
population, then performance may be affected by population or disease
prevalence bias.’ They continue: ‘It is thus advised to use images from
multiple diverse sources, or at least images representing the target
population or health system in which the algorithm is to be deployed.’ What is
evidenced here is how the explanatory link between algorithmic biases and
biases in training data informs algorithmic bias mitigation strategies. In
particular, avoiding demographic performance disparities typically requires
ensuring that training data appropriately represents the target population.

##  3 The Generalist Paradigm

The main commitment of the narrow paradigm in clinical machine learning is
that models are trained for particular clinical use cases such as risk
assessment, diagnosis and treatment recommendation. The emerging generalist
paradigm differs in that its core focus is on training generalist models that
can be adapted for multiple clinical use cases via fine-tuning on narrow
biomedical datasets. This section introduces the generalist paradigm and
examines in what respects this paradigm challenges the ontic and explanatory
assumptions about algorithmic bias in medicine.

###  3.1 Self-Supervised Learning

The models with which we are concerned are large language models such as Open
AI’s GPT-2 and GPT-3 (Radford et al., 2019, Brown et al., 2020), Google’s BERT
and LaMDA (Devlin et al., 2018, Thoppilan et al., 2022), and Meta’s OPT (Zhang
et al., 2022). These models are pre-trained for natural language
understanding, and can be fine-tuned on domain-specific datasets for
particular use cases including clinical use cases. To illustrate: Initially,
BERT is pre-trained on English Wikipedia and BooksCorpus for natural language
understanding (Devlin et al., 2018, p.5). BERT can then be fine-tuned for
specific clinical tasks, such as biomedical text mining, through further
training on biomedical text datasets such as PubMed (c.f. Lee et al., 2020).

Language models are pre-trained for natural language understanding using a
machine learning technique called self-supervised learning (Bommasani et al.,
2021, pp.4-5). Recall that supervised learning involves labeled data. For
example, a supervised model for named entity recognition might be trained on
ordered pairs containing a biomedical noun and the type of entity to which
each noun refers, e.g. (TP53, gene), (thrombin, enzyme). In contrast, self-
supervised models are trained on unlabelled text datasets like English
Wikipedia (Devlin et al., 2018). Training tasks on unlabelled text data are
automated prediction tasks. For example, one of BERT’s training tasks is
called masked language modeling, in which particular words from sentences in
the training data are masked such that the model must predict the missing
words based on the surrounding words. This task teaches BERT bidirectional
context. Given text datasets with the scale and breadth of English Wikipedia,
masked language modeling allows language models to learn a broad class of
capabilities including arithmetic (‘100 x 20 =  ’), general knowledge (‘ is
the capital of France), specialist knowledge (‘The $LD_{50}$ of ricin for mice
is  mg/kg’), translation (‘Ospedale is the Italian word for  ’), and unit
conversion (‘The adult spleen is approximately 14cm ( inches) in length’).
Hence the generalist character of these models.

To be precise: What is learned in the pre-training stage is a mathematical
representation of a language called a word embedding. Roughly, a word
embedding is a function that maps words to vectors (or points) in a high-
dimensional space.555This formulation is rough because strictly speaking
tokens and not words are inputted into the function, i.e. input sequences of
text are first tokenized (see Grefenstette, 1999). The geometry of the vector
space encodes semantic information in that distances between vectors represent
the degree to which words are semantically related. Words with similar
meanings map to vectors which are close together. The large language models
that we are considering complicate this picture slightly. Notice that certain
words have multiple meanings. For example, ‘bank’ can mean the side of a river
or it can mean a type of financial institution. Hence it is problematic to map
each word to a vector that encodes its meaning, as some words do not have one
meaning. Large language models learn what is called a contextual embedding,
namely, a function that maps each word in an input sequence to a vector in a
high-dimensional space, where the vector’s location depends on the entire
input sequence as opposed to individual words (Liu et al., 2020). This allows
the model to discriminate between different senses of the same word.

Fine-tuning for clinical use cases differs depending on the pre-trained
model’s architecture. It is helpful to distinguish two cases.666A third case
bracketed here is sequence-to-sequence models such as Google’s T5 that include
an encoder and a decoder (Raffel et al., 2020). First, encoder models such as
BERT take text sequences as inputs and output embeddings for each inputted
word. Fine-tuning these models for clinical uses typically involves additional
pre-training on biomedical text datasets such as PubMed abstracts and MIMIC
clinical notes (Peng et al., 2019). The result is an embedding that better
registers the semantic nuances of biomedical text data; although in practice
whether domain-specific pre-training improves downstream model performance
depends on the task at issue (Lin et al., 2020). Biomedical embeddings can
then be used as inputs for supervised models that are trained for particular
clinical use cases. To illustrate: Suppose the task at issue is hospital
readmission prediction using free-text discharge summaries (c.f. Huang et al.,
2019). Given a labeled dataset comprising discharge summaries plus the ground
truth for whether patients were readmitted within the relevant time-window,
the idea is to train a supervised model to predict readmission using
embeddings for the discharge summaries as inputs.

Second, decoder models like GPT-2 (Radford et al., 2019) and LaMDA (Thoppilan
et al., 2022) take sequences of text as their inputs and output a prediction
for the next word. Feeding the outputs back into the model allows passages of
text to be generated. Fine-tuning decoder models for clinical use cases
involves next-word prediction tasks on tailored biomedical text datasets. For
example, Wang et al. (2021) fine-tuned GPT-2 using transcripts of therapy
sessions to evidence the in principle use of language models in therapeutic
contexts. Likewise, Sirrianni et al. (2022) fine-tuned GPT-2 on dental
clinical notes to evidence the in principle use of language models for medical
charting. In addition to fine-tuning, there is also promising evidence
suggesting that pre-trained language models that have not been fine-tuned on
domain-specific clinical data encode clinical knowledge (Singhal et al.,
2022). The idea is that appropriately tailored prompts can enable the model to
accurately answer medical questions.

###  3.2 Semantic Biases

The ontic assumption holds that algorithmic biases in clinical medicine admit
analysis in terms of measurable performance disparities across demographic
groups. Self-supervised models challenge this conception of algorithmic bias
by introducing uniquely semantic biases that are non-reducible to measurable
demographic performance disparities. To be clear, semantic biases are
understood as a subset of algorithmic biases. What makes these biases semantic
is that the nature of these biases has to do with meaning. Consider three
examples.777These examples are illustrative and are not intended as an
exhaustive taxonomy.

Call the first kind of semantic bias perspectival bias. What is at issue here
is that evaluative commitments may be implicit in the way that words are used
or defined. Weidinger et al. (2021, p.13) give an example in which a language
model asked ‘What is a family?’ responds ‘A family is a man and a woman who
get married and have children.’ Here the meaning attached to the word ‘family’
reflects a heteronormative conception of the family, which is incompatible
with conceptions of the family that, for example, make room for families
involving same-sex couples or polyamorous familial relationships involving
more than two partners.

Perspectival biases are morally significant in clinical use cases for at least
two reasons. First, patient-facing conversational interfaces such as
therapeutic chatbots or question answering systems may alienate particular
patient subpopulations by exhibiting perspectival biases that diminish or deny
the significance of certain beliefs, values, identities, or cultural
practices. For example, a therapeutic chatbot that indicates a commitment to a
hetronormative conception of the family may alienate LGBTQ+ patients and
engender the perception that clinical machine learning tools are not developed
with LGBTQ+ individuals in mind. What is concerning here is that perspectival
biases may be difficult to anticipate in practice and need not involve any
explicit diminution of beliefs, values, identities, or practices. For example,
an entertainment chatbot on a pediatric ward may ask children whether they are
missing home. If pressed to clarify the question, the chatbot may associate
‘home’ with a static residence, which may be alienating for children in the
Irish Traveller community or other nomadic communities. Or consider a sexual
health chatbot that when pressed to define or elaborate upon terms relevant to
gender identity does so using pathologizing or otherwise stigmatizing
terminology.

Second, dialogue-based health information systems may exhibit undue influence
over the patient’s decision if certain outputs are loaded with perspectival
bias. The concern is that a system’s explanation of the costs and benefits of
particular treatments may fail to be value neutral in such a way that amounts
to the inappropriate distortion of information material to the patient’s
decision. This could happen if, for example, qualitative categories reflecting
the probability of outcomes (e.g. ‘likely’, ‘unlikely’) or the severity of
outcomes (‘severe’, ‘mild’) are used by chatbots to communicate risks of
particular interventions (McNeil et al., 1982, Halpern et al., 2013). To
illustrate: The claim that Pre-Exposure Prophylaxis (PrEP) for HIV is likely
to cause a mild reduction in kidney function may be interpreted by the patient
as the claim that the risk can justifiably be ignored. Yet whether the risk
can justifiably be ignored depends on the patient’s risk tolerance and the
anticipated benefit of PrEP for them given their beliefs and values. Hence the
usage of these qualitative risk descriptions may inadvertently impose a
particular perspective on the risks of PrEP that inappropriately biases the
patient’s decision.

Call the second class of semantic biases stereotype associations. To
illustrate: In one recent study, Abid et al. (2021) found that in 66/100 cases
GPT-3 completed the prompt ‘two Muslims walked into a’ with violent
phraseology including references to killing and shooting. Relevantly analogous
stereotype associations have been observed for LGBTQ individuals, alongside
women and people of color Sheng et al. (2019). Roughly speaking, the technical
explanation for these stereotype biases is that the model infers semantic
dependence between, for example, the terms Muslim and terrorism due to
frequent co-occurrence of the terms in the training data (Brown et al., 2020).
What is concerning about the possibility of language models exhibiting
stereotype associations in conversational interfaces for clinical use cases is
the idea that clinical technologies may in some way exacerbate existing stigma
and clinical harm faced by marginalized populations.

Call the third kind of semantic bias semantic haze. Some discourses can be
better represented than others in the internet text datasets such that
language models are pre-trained on. It may be the case, for example, that
discourse on the United States is more prevalent than discourse on Honduras on
particular Reddit datasets, i.e. there is more text about the United States
than Honduras on the relevant Reddit pages. That certain discourses feature
less prominently as part of text datasets can result in language models having
a coarser-grained or ‘hazy’ semantic representation of those discourses. For
example, in BERT’s embedding space, the term ‘United States’ is a greater
distance from the names of other countries (and thus more semantically
distinct) compared to Global South countries which are clustered together and
thus less semantically distinct. BERT has a finer-grained semantic
representation of discourse pertaining to the United States than to Global
South countries (Zhou et al., 2021, pp.7-8).

The ethical significance of semantic haze in clinical use cases is at least
threefold. First, there is a clear sense in which patient-facing
conversational interfaces that exhibit semantic haze on particular topics may
perform less well for some patient groups compared to others. Consider a
chatbot that provides sex and relationship information for teenagers. The
kinds of questions and concerns that teenagers have around sex and
relationships may depend at least in part on cultural circumstances. Indeed,
some concerns may require specific cultural knowledge to understand and
provide appropriate responses to. Suppose, for example, that a 16-year-old
female from a Christian family is engaged in a sexually active relationship
with a 15-year old male from an Islamic family. There are concerns within the
male’s family around premarital sex, and the family is suggesting to the
female to make nikah (the Islamic marriage contract), and potentially to take
the Shahada (a declaration of belief in God and acceptance of Muhammad as
God’s messanger). Semantic haze around Islamic marriage customs given limited
exposure to text on Islamic marriage customs in the pre-training dataset may
result in suboptimal model performance with respect to providing sex and
relationships advice on cases like these. A related issue is that language
models may be more likely to hallucinate false information in domains of
discourse that are underrepresented in the training data (Ji et al., 2023).
For some clinical use cases, this may result in lower health information
quality for certain groups.

Second, the moral significance of semantic haze is also at least in part
accounted for in terms of respect for persons. To illustrate: A known problem
in palliative care is that a non-trivial fraction of patients feel ignored,
misunderstood, or even silenced by clinicians (Norton et al., 2003, Frosch et
al., 2012, Gramling et al., 2016). This phenomenon is morally significant not
only because it causes patients to experience psychological discomfort, but
also because failing to listen to and appropriately register patient wishes
and concerns plausibly amounts to a failure to afford due respect to patients
as persons. There are plausible use cases for language models in palliative
care including dialogue agents for companionship (van Wezel et al., 2021) and
entertainment (García-Méndez et al., 2021). Here it is at least foreseeable
that models exhibiting semantic haze around certain topics may engender a
sense of not being heard on the part of patients, such that deployment of
models exhibiting the relevant semantic haze may amount to a failure to afford
due respect to the patient. Indeed the problem is not restricted to palliative
care. Minority patient groups such as transgender patients or patients with
disabilities may have a pre-existing sense of being misunderstood or silenced
in the healthcare system, and conversational systems exhibiting semantic haze
around health topics specific to those groups may invite a sense of their
person or their identity having been disrespected among members of the
relevant minority groups.

Third, in practice, it may be the case that models which perform poorly or
unpredictably for certain groups are not made available to those groups or are
subject to certain restrictions on how the models can be used that are not
applied to other groups. Consider an online medical question answering tool
that reliably answers questions in English but not Spanish. Such a tool may
screen prompts with a language classifier so as to detect and refuse prompts
written in Spanish. In doing so, the potential benefits of the tool for
Spanish speaking users who do not speak English are withheld because the
safety risk to that group is intolerably high. Obviously, product decisions
like these may compound across multiple tools resulting in patterns of
inequity with respect to health information availability across demographic
groups.

###  3.3 New Explanatory Pathways

The last section showed how language models may introduce novel and uniquely
semantic biases. These biases matter for conversational use cases such as
therapeutic chatbots and question-answering systems. But their significance
for non-conversational use cases such as risk assessment or diagnosis based on
clinical notes is less apparent. This section argues that in non-
conversational use cases language models can complicate the explanatory
relationship between biases in training data and familiar model performance
biases such as disparities in accuracy across demographic groups. This
complication is significant because it destabilizes the standard bias
mitigation strategy of ensuring appropriately representative training data.

Semantic associations learned in pre-training may in some cases partly explain
performance disparities in downstream clinical prediction tasks. Consider an
example. Suppose that BERT is fine-tuned for the psychiatric use case of EHR-
based violence risk assessment (Mosteiro et al., 2022, Karystianis et al.,
2021). In pre-training BERT will have learned semantic associations between
terms like ‘violence,’ ‘resist,’ and ‘non-compliance,’ and these associations
may inform the model’s predictions in the downstream task. The presence of
such terms under the right contextual circumstances may be good predictors of
violence. Indeed, the principal advantage of BERT-based models for EHR-based
violence risk assessment is that BERT embeddings are sensitive to context.
Hence the expectation is that semantic associations learned in pre-training
affect and improve performance in the downstream clinical task.

However, there is evidence that negative descriptors such as ‘resist’ and
‘non-compliant’ are more likely to feature in EHRs for Black patients compared
to non-Black patients (Sun et al., 2022). Suppose for the purposes of the
example that a racial disparity in the use of negative descriptors obtains and
is at least partly explained by implicit racial biases on the part of
clinicians.888Sun et al. (2022, p.206) note that ‘the use of negative
descriptors might not necessarily reflect bias among individual providers;
rather, it may reflect a broader systemic acceptability of using negative
patient descriptors as a surrogate for identifying structural barriers.’ To
clarify: What is being assumed here is that negative descriptors are in fact
used more for Black patients compared to non-Black patients and that the
explanation for the disparity is that clinicians as a group are disposed at
least to some extent to use more negative language when describing
interactions with Black patients compared to relevantly similar interactions
involving non-Black patients. Given these assumptions, it may be the case that
a fine-tuned BERT model for violence risk assessment ends up overpredicting
violence for Black patients because the kinds of negative language that best
predict violent behavior are disproportionately used to characterize
interactions with Black patients. More precisely, the descriptive bias in the
EHR data may result in a failure of calibration across racial groups, such
that for any given risk score the fraction of actually violent Black patients
at that risk score is greater than the fraction of actually violent non-Black
patients at that same risk score (Hedden, 2021, Kleinberg et al., 2016).

What explains the calibration failure in the above example is
misrepresentative training data. Yet the way in which the training data is
misrepresentative is novel. What is not at issue is a proxy variable that
misrepresents the circumstances of Black patients - as was the case in the
Obermeyer et al. (2019) example in which the model used health cost as a proxy
for health need. Neither is data missing or mislabelled. Rather the problem is
that (by assumption) free-text descriptions of Black patients in EHRs are
disproportionately negative in the sense of exaggerating the degree of non-
adherence in contrast to non-Black patients who display comparable levels of
non-adherence. Language models can register differences in the language used
to characterize different demographic groups. Biases encoded in those textual
descriptions may translate into model performance biases in a way that is hard
to detect.

Two points are made in closing: (1) It is plausible that clinicians harbor
implicit or even explicit biases against disadvantaged groups including racial
minorities, people with disabilities, and transgender individuals (see Hall et
al. (2015) for a review). Such biases may affect clinical notes written about
such patients in respects that are not yet well understood and in respects
that language models may register. (2) This complicates the standard bias
mitigation advice to ensure that clinical training data is representative of
the target population.999Note that this issue is not unique to LLMs. Similar
considerations hold for models and research studies that rely on discrete EHR
data, which can also encode biases (c.f. Ross et al., 2020). For it may be
unclear in what respects textual data fails to appropriately represent
disadvantaged subpopulations. In particular, where negative leaning language
is involved or pathologizing language is used to characterize patient
concerns.

##  4 Mitigating Bias

The emerging generalist paradigm in clinical machine learning complicates the
prevailing ontic assumption that algorithmic biases admit analysis in terms of
performance disparities across demographic groups. The generalist paradigm
also puts question to the prevailing explanatory assumption that biases in
model performance mirror biases in training datasets such that bias mitigation
strategies ought to focus on demographic representation in biomedical
datasets. The upshot is that regulatory policy developed on the backdrop of
the narrow paradigm - such as the FDA’s recommendation that ‘[d]ata collection
protocols should ensure that the relevant characteristics of the intended
patient population […] are sufficiently represented in […] test datasets so
that results can reasonably be generalized to the population of interest’ -
has limited practical significance for clinical developers using generalist
models (Food and Drug Administration, 2021a, p.2).

To be sure, biomedical datasets have typically been used to fine-tune
generalist models for clinical use cases; and to that end, ensuring
appropriate representation within the datasets used for fine-tuning is
actionable and relevant advice. Yet two caveats are needed. On one hand, bias
mitigation guidance that focuses exclusively on model performance biases that
are explained by biases in biomedical datasets occludes considerations
relevant to bias mitigation in generalist models. In particular, performance
biases in clinical models may be explained by biases in pre-training data
which is not specifically biomedical, so there is good reason not to provide
bias mitigation guidance which pertains uniquely to biomedical data. On the
other hand, there now exist promising examples of clinical information
extraction from generalist models that are not fine-tuned on biomedical data
(Singhal et al., 2022). Bias guidance that focuses uniquely on the
representativeness of biomedical data does not apply to these models and thus
excludes an important emerging class of clinical machine learning systems.

What follows are some plausible directions for bias mitigation guidance in the
context of generalist language models applied to clinical use cases.

###  4.1 Pre-Processing and Data Collection

The first bias mitigation strategy targets the data pre-processing and data
collection stages. Pre-processing is an umbrella term for a set of data
practices including cleansing datasets for incorrect or missing data and
wrangling raw data into an appropriate format for machine learning
development. Internet text datasets can be filtered for inappropriate content
in the pre-processing stage. The Colossal Clean Crawled Corpus, for example,
is a 156 billion token text dataset that has been filtered using the open
access ‘Dirty, Naughty, Obscene or Otherwise Bad Words List’ (Raffel et al.,
2020). The rationale behind filtering text datasets for inappropriate content
is that language models are unable to reproduce the kinds of harmful
stereotype associations and perspectival biases that arise in, for example,
pornographic websites or websites containing far-right political content, if
all such websites are removed from the model’s training corpus (Bender et al.,
2021, p.614). Hence data filtration is a plausible approach to mitigating
semantic biases that may arise in conversational clinical use cases such as
therapeutic chatbots.

Nevertheless, filtering text datasets for problematic content involves trade-
offs that require articulation and judicious negotiation. On one hand,
removing toxic language from training data precludes models from identifying
situations in which they are confronted with toxic language and identifying
situations in which the model’s own outputs are toxic. Hence data filtration
may actually hinder the model’s ability to counteract biases. On the other
hand, data filtration can also result in semantic haze around marginalized
discourses. Bender et al. (2021, p.614) discuss one example in which the term
‘twink’ features on the ‘Dirty, Naughty, Obscene or Otherwise Bad Words List,’
such that text datasets filtered by this list may systematically exclude
LGBTQ+ online spaces. Such exclusion may render the model less able to
register semantic distinctions and draw appropriate semantic associations for
LGBTQ+ terminology (Zhou et al., 2021), which may be problematic in patient-
facing conversational use cases such as therapeutic chatbots and question-
answering systems. These concerns underscore the importance of active dataset
diversification efforts as an important complementary technique alongside
filtration for problematic content.

In addition to these concerns there is a more fundamental worry about who is
best placed to decide which content is appropriate for text datasets used to
train language models that are subsequently adapted for clinical use cases.
These assumptions are potentially problematic in clinical use cases in that
patients and clinicians represent a plurality of worldviews and there is an
expectation of reasonable disagreement between stakeholders about what counts
as appropriate content. While there are plausible mitigations such as
accommodating a diversity of cultural backgrounds and political opinions in
data filtration decisions, the question of how to establish such an effort in
a way that is politically legitimate admits no easy answer.

###  4.2 Fine-Tuning

The second bias mitigation strategy is to fine-tune the model to avoid harmful
or otherwise problematic content. For example, Google’s LaMDA is fine-tuned
using supervised learning to ensure appropriate model outputs (Thoppilan et
al., 2022). In particular, crowdworkers were used to generate conversational
material with LaMDA including conversations on sensitive topics and
adversarial conversations in which crowd workers attempted to elicit
inappropriate model outputs. These outputs were subsequently annotated to
identify problematic content, and also the type of problematic content at
issue (e.g. religious stereotypes). The resultant dataset was used to fine-
tune LaMDA to predict the appropriateness of its outputs which in turn allowed
LaMDA to screen-out problematic outputs.101010Ethical fine-tuning can also be
achieved via non-supervised approaches such as Reinforcement Learning from
Human Feedback (Bai et al., 2022, Ouyang et al., 2022). The ethical issues
discussed in this section apply to these non-supervised approaches also.

Ethical fine-tuning is a promising technique for addressing semantic biases in
clinical use cases including stereotype associations and perspectival biases.
Certain clinical use cases require the model to engage with sensitive content
and do so appropriately, such as therapeutic use cases in which the model may
need to engage in discussions about suicide, drug use, or sex. This gives
ethical fine-tuning the advantage over data filtration. Removing a chatbot’s
ability to discuss sex by filtering all sexual content from the training data
caps the therapeutic utility of the chatbot substantially. In contrast,
ethical fine-tuning offers a plausible avenue for chatbots to engage in
dialogue around sensitive topics and to do so appropriately. Presumably,
mental health professionals as opposed to crowdworkers would be required to
construct and label training data involving appropriate and inappropriate
therapeutic conversational interactions. This may render ethical fine-tuning
for clinical use cases costly given the need for expert clinical labelers. But
what matters here is that ethical fine-tuning provides a plausible and
scalable mechanism for automated appropriateness checks on clinical model
outputs.

There are, however, at least two limitations to fine-tuning. First, using
human annotators to discover failure modes in which the model exhibits harmful
biases limits the diversity of failure modes that can be accounted for (Perez
et al., 2022). Because clinical models can reasonably be expected to encounter
a much broader class of bias-eliciting stimuli in the real-world deployment
context, it is at least plausible that fine-tuning based on human-labeled
dialogical interactions will lead to critical oversights. The practical
significance of this concern depends in large part on the success or failure
of emerging strategies for upscaling ethical fine-tuning via automated case
generation (Perez et al., 2022, Ganguli et al., 2022), and in particular on
the applicability of these techniques to conversational clinical use cases.
Second, the capacity of generalist language models to exhibit biases is
virtually unbounded, especially with respect to perspectival biases which are
pervasive in internet text corpora. For that reason it is unclear if the
appropriate goal of ethical fine-tuning ought to be perspectival neutrality or
rather the adoption of a perspective that is broadly acceptable within the
bounds of a pluralistic society. The issue here is that ethical fine-tuning is
not itself a solution to the problem of semantic biases in clinical models,
but rather a technical method that if supplemented with an appropriate set of
evaluative commitments can address the problem. The questions of who is doing
the fine-tuning and by what criteria cannot be overlooked. This underscores
the need for inclusive and open discussion around how ethical fine-tuning is
performed in practice.

##  5 Conclusion

This paper examined how certain pervasive assumptions about the nature and
causes of algorithmic bias in medicine are challenged by an emerging class of
generalist models such as BERT and GPT-2 that can be fine-tuned for clinical
use cases. Specifically, medical regulators such as the FDA in the US and the
MHRA in the UK are operating with a conception of bias that is tailored to the
dominant narrow paradigm in clinical machine learning, in which supervised
models are trained to perform particular tasks such as diagnosis and risk
assessment. On this picture, models exhibit biases if and to the extent that
they admit measurable performance disparities across demographic groups such
as differences in classification accuracy or miscalibrated risk scores. These
biases are typically explained by analogous biases in training data such as
underrepresentation or misrepresentation of the relevant demographic groups.
Generalist models complicate this picture. Not only do these models introduce
novel and uniquely semantic biases such as stereotype associations and
perspectival biases, they also complicate the explanatory relationship between
biases in training data and measurable performance disparities across
demographic groups for regression and classification tasks.

Three recommendations are made in closing. First, there is good reason for
SaMD medical regulators to explore emerging techniques for bias mitigation in
large language models such as data filtration and ethical fine-tuning and to
incorporate such techniques into guidance and regulatory policy on the safe
and ethical development of clinical machine learning models (Bender et al.,
2021, Laurençon et al., 2022). Second, healthcare providers seeking to use
generalist models for clinical applications ought to consider algorithmic bias
not only from the perspective of performance disparities across demographic
groups relative to regression and classification metrics, but also consider
auditing models for clinically significant semantic biases. Third, medical
ethicists have good reason to examine the moral significance of biases in
clinical models beyond considerations pertaining to the equitable distribution
of benefits and burdens in healthcare. For example, examining the class of
respect-based moral complaints that patients may have towards models
exhibiting biases that reflect harmful stereotype associations.

## Acknowledgements

I am grateful to Michael Howell, Heather Cole-Lewis, Lisa Lehmann, Diane
Korngiebel, Thomas Douglas, Bakul Patel, Kate Weber, and Rachel Gruner for
helpful comments, alongside participants at the Workshop on the Ethics of
Influence at the Uehiro Centre for Practical Ethics at the University of
Oxford.

## References

  * Abid et al. [2021] A. Abid, M. Farooqi, and J. Zou.  Persistent anti-muslim bias in large language models.  In _Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society_ , pages 298–306, 2021.
  * Bai et al. [2022] Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al.  Training a helpful and harmless assistant with reinforcement learning from human feedback.  _arXiv preprint arXiv:2204.05862_ , 2022.
  * Bender et al. [2021] E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell.  On the dangers of stochastic parrots: Can language models be too big?  In _Proceedings of the 2021 ACM conference on fairness, accountability, and transparency_ , pages 610–623, 2021.
  * Bommasani et al. [2021] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv:2108.07258_ , 2021.
  * Brown et al. [2020] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Challen et al. [2019] R. Challen, J. Denny, M. Pitt, L. Gompels, T. Edwards, and K. Tsaneva-Atanasova.  Artificial intelligence, bias and clinical safety.  _BMJ Quality & Safety_, 28(3):231–237, 2019\.
  * Chouldechova [2017] A. Chouldechova.  Fair prediction with disparate impact: A study of bias in recidivism prediction instruments.  _Big data_ , 5(2):153–163, 2017.
  * Chowdhury et al. [2021] A. Chowdhury, J. Rosenthal, J. Waring, and R. Umeton.  Applying self-supervised learning to medicine: review of the state of the art and medical implementations.  In _Informatics_ , volume 8(3), page 59. MDPI, 2021.
  * Cirillo et al. [2020] D. Cirillo, S. Catuara-Solarz, C. Morey, E. Guney, L. Subirats, S. Mellino, A. Gigante, A. Valencia, M. J. Rementeria, A. S. Chadha, et al.  Sex and gender differences and biases in artificial intelligence for biomedicine and healthcare.  _NPJ digital medicine_ , 3(1):81, 2020.
  * Daneshjou et al. [2022] R. Daneshjou, K. Vodrahalli, R. A. Novoa, M. Jenkins, W. Liang, V. Rotemberg, J. Ko, S. M. Swetter, E. E. Bailey, O. Gevaert, et al.  Disparities in dermatology ai performance on a diverse, curated clinical image set.  _Science advances_ , 8(31):eabq6147, 2022.
  * Deng et al. [2009] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–255. Ieee, 2009.
  * Devlin et al. [2018] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Dieterich et al. [2016] W. Dieterich, C. Mendoza, and T. Brennan.  Compas risk scales: Demonstrating accuracy equity and predictive parity.  _Northpointe Inc_ , 7(7.4):1, 2016.
  * Food and Drug Administration [2021a] Food and Drug Administration.  Artificial intelligence/machine learning (ai/ml)-based software as a medical device (samd) action plan.  _Food Drug Admin., Silver Spring, MD, USA, Tech. Rep_ , 1, 2021a.
  * Food and Drug Administration [2021b] Food and Drug Administration.  Good machine learning practice for medical device development: Guiding principles, 2021b.
  * Frosch et al. [2012] D. L. Frosch, S. G. May, K. A. Rendle, C. Tietbohl, and G. Elwyn.  Authoritarian physicians and patients’ fear of being labeled ‘difficult’among key obstacles to shared decision making.  _Health affairs_ , 31(5):1030–1038, 2012.
  * Ganguli et al. [2022] D. Ganguli, L. Lovitt, J. Kernion, A. Askell, Y. Bai, S. Kadavath, B. Mann, E. Perez, N. Schiefer, K. Ndousse, et al.  Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned.  _arXiv preprint arXiv:2209.07858_ , 2022.
  * García-Méndez et al. [2021] S. García-Méndez, F. De Arriba-Pérez, F. J. González-Castaño, J. A. Regueiro-Janeiro, and F. Gil-Castiñeira.  Entertainment chatbot for the digital inclusion of elderly people without abstraction capabilities.  _IEEE Access_ , 9:75878–75891, 2021.
  * Gianattasio et al. [2019] K. Z. Gianattasio, C. Prather, M. M. Glymour, A. Ciarleglio, and M. C. Power.  Racial disparities and temporal trends in dementia misdiagnosis risk in the united states.  _Alzheimer’s & Dementia: Translational Research & Clinical Interventions_, 5:891–898, 2019.
  * Gianfrancesco et al. [2018] M. A. Gianfrancesco, S. Tamang, J. Yazdany, and G. Schmajuk.  Potential biases in machine learning algorithms using electronic health record data.  _JAMA internal medicine_ , 178(11):1544–1547, 2018.
  * Gramling et al. [2016] R. Gramling, S. Stanek, S. Ladwig, E. Gajary-Coots, J. Cimino, W. Anderson, S. A. Norton, R. A. Aslakson, K. Ast, R. Elk, et al.  Feeling heard and understood: a patient-reported quality measure for the inpatient palliative care setting.  _Journal of pain and symptom management_ , 51(2):150–154, 2016.
  * Grefenstette [1999] G. Grefenstette.  Tokenization.  _Syntactic Wordclass Tagging_ , pages 117–133, 1999.
  * Groh et al. [2021] M. Groh, C. Harris, L. Soenksen, F. Lau, R. Han, A. Kim, A. Koochek, and O. Badri.  Evaluating deep neural networks trained on clinical images in dermatology with the fitzpatrick 17k dataset.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1820–1828, 2021.
  * Grote and Keeling [2022a] T. Grote and G. Keeling.  On algorithmic fairness in medical practice.  _Cambridge Quarterly of Healthcare Ethics_ , 31(1):83–94, 2022a.
  * Grote and Keeling [2022b] T. Grote and G. Keeling.  Enabling fairness in healthcare through machine learning.  _Ethics and Information Technology_ , 24(3):39, 2022b.
  * Hall et al. [2015] W. J. Hall, M. V. Chapman, K. M. Lee, Y. M. Merino, T. W. Thomas, B. K. Payne, E. Eng, S. H. Day, and T. Coyne-Beasley.  Implicit racial/ethnic bias among health care professionals and its influence on health care outcomes: a systematic review.  _American journal of public health_ , 105(12):e60–e76, 2015.
  * Halpern et al. [2013] S. D. Halpern, G. Loewenstein, K. G. Volpp, E. Cooney, K. Vranas, C. M. Quill, M. S. McKenzie, M. O. Harhay, N. B. Gabler, T. Silva, et al.  Default options in advance directives influence how patients set goals for end-of-life care.  _Health Affairs_ , 32(2):408–417, 2013.
  * Hasan et al. [2010] O. Hasan, D. O. Meltzer, S. A. Shaykevich, C. M. Bell, P. J. Kaboli, A. D. Auerbach, T. B. Wetterneck, V. M. Arora, J. Zhang, and J. L. Schnipper.  Hospital readmission in general medicine patients: a prediction model.  _Journal of general internal medicine_ , 25:211–219, 2010\.
  * Hedden [2021] B. Hedden.  On statistical criteria of algorithmic fairness.  _Philosophy and Public Affairs_ , 49(2):209–231, 2021.
  * Hellström et al. [2020] T. Hellström, V. Dignum, and S. Bensch.  Bias in machine learning–what is it good for?  _arXiv preprint arXiv:2004.00686_ , 2020.
  * Huang et al. [2019] K. Huang, J. Altosaar, and R. Ranganath.  Clinicalbert: Modeling clinical notes and predicting hospital readmission.  _arXiv preprint arXiv:1904.05342_ , 2019.
  * Ji et al. [2023] Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. J. Bang, A. Madotto, and P. Fung.  Survey of hallucination in natural language generation.  _ACM Computing Surveys_ , 55(12):1–38, 2023.
  * Jiang and Nachum [2020] H. Jiang and O. Nachum.  Identifying and correcting label bias in machine learning.  In _International Conference on Artificial Intelligence and Statistics_ , pages 702–712. PMLR, 2020.
  * Karystianis et al. [2021] G. Karystianis, R. C. Cabral, S. C. Han, J. Poon, and T. Butler.  Utilizing text mining, data linkage and deep learning in police and health records to predict future offenses in family and domestic violence.  _Frontiers in digital health_ , 3:602683, 2021.
  * Kelly et al. [2022] B. S. Kelly, C. Judge, S. M. Bollard, S. M. Clifford, G. M. Healy, A. Aziz, P. Mathur, S. Islam, K. W. Yeom, A. Lawlor, et al.  Radiology artificial intelligence: a systematic review and evaluation of methods (raise).  _European radiology_ , 32(11):7998–8007, 2022\.
  * Kleinberg et al. [2016] J. Kleinberg, S. Mullainathan, and M. Raghavan.  Inherent trade-offs in the fair determination of risk scores.  _arXiv preprint arXiv:1609.05807_ , 2016.
  * Laurençon et al. [2022] H. Laurençon, L. Saulnier, T. Wang, C. Akiki, A. Villanova del Moral, T. Le Scao, L. Von Werra, C. Mou, E. González Ponferrada, H. Nguyen, et al.  The bigscience roots corpus: A 1.6 tb composite multilingual dataset.  _Advances in Neural Information Processing Systems_ , 35:31809–31826, 2022.
  * Lee et al. [2020] J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. H. So, and J. Kang.  Biobert: a pre-trained biomedical language representation model for biomedical text mining.  _Bioinformatics_ , 36(4):1234–1240, 2020.
  * Lin et al. [2020] C. Lin, S. Bethard, D. Dligach, F. Sadeque, G. Savova, and T. A. Miller.  Does bert need domain adaptation for clinical negation detection?  _Journal of the American Medical Informatics Association_ , 27(4):584–591, 2020.
  * Liu et al. [2020] Q. Liu, M. J. Kusner, and P. Blunsom.  A survey on contextual embeddings.  _arXiv preprint arXiv:2003.07278_ , 2020.
  * McNeil et al. [1982] B. J. McNeil, S. G. Pauker, H. C. Sox Jr, and A. Tversky.  On the elicitation of preferences for alternative therapies.  _New England journal of medicine_ , 306(21):1259–1262, 1982.
  * Mitsios et al. [2018] J. P. Mitsios, E. I. Ekinci, G. P. Mitsios, L. Churilov, and V. Thijs.  Relationship between glycated hemoglobin and stroke risk: a systematic review and meta-analysis.  _Journal of the American Heart Association_ , 7(11):e007858, 2018.
  * Mosteiro et al. [2022] P. Mosteiro, E. Rijcken, K. Zervanou, U. Kaymak, F. Scheepers, and M. Spruit.  Machine learning for violence risk assessment using dutch clinical notes.  _arXiv preprint arXiv:2204.13535_ , 2022.
  * Norori et al. [2021] N. Norori, Q. Hu, F. M. Aellen, F. D. Faraci, and A. Tzovara.  Addressing bias in big data and ai for health care: A call for open science.  _Patterns_ , 2(10):100347, 2021.
  * Norton et al. [2003] S. A. Norton, V. P. Tilden, S. W. Tolle, C. A. Nelson, and S. T. Eggman.  Life support withdrawal: communication and conflict.  _American Journal of Critical Care_ , 12(6):548–555, 2003.
  * Obermeyer et al. [2019] Z. Obermeyer, B. Powers, C. Vogeli, and S. Mullainathan.  Dissecting racial bias in an algorithm used to manage the health of populations.  _Science_ , 366(6464):447–453, 2019.
  * Ouyang et al. [2022] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744, 2022.
  * Panch et al. [2019] T. Panch, H. Mattie, and R. Atun.  Artificial intelligence and algorithmic bias: implications for health systems.  _Journal of global health_ , 9(2), 2019.
  * Peng et al. [2019] Y. Peng, S. Yan, and Z. Lu.  Transfer learning in biomedical natural language processing: an evaluation of bert and elmo on ten benchmarking datasets.  _arXiv preprint arXiv:1906.05474_ , 2019.
  * Perez et al. [2022] E. Perez, S. Huang, F. Song, T. Cai, R. Ring, J. Aslanides, A. Glaese, N. McAleese, and G. Irving.  Red teaming language models with language models.  _arXiv preprint arXiv:2202.03286_ , 2022.
  * Radford et al. [2019] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9, 2019.
  * Raffel et al. [2020] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):5485–5551, 2020.
  * Rahimi et al. [2021] S. Rahimi, O. Oktay, J. Alvarez-Valle, and S. Bharadwaj.  Addressing the exorbitant cost of labeling medical images with active learning.  In _International Conference on Machine Learning in Medical Imaging and Analysis_ , page 1, 2021.
  * Rajkomar et al. [2018] A. Rajkomar, M. Hardt, M. D. Howell, G. Corrado, and M. H. Chin.  Ensuring fairness in machine learning to advance health equity.  _Annals of internal medicine_ , 169(12):866–872, 2018.
  * Rajkomar et al. [2019] A. Rajkomar, J. Dean, and I. Kohane.  Machine learning in medicine.  _New England Journal of Medicine_ , 380(14):1347–1358, 2019.
  * Rasmy et al. [2021] L. Rasmy, Y. Xiang, Z. Xie, C. Tao, and D. Zhi.  Med-bert: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction.  _NPJ digital medicine_ , 4(1):86, 2021.
  * Ross et al. [2020] A. B. Ross, V. Kalia, B. Y. Chan, and G. Li.  The influence of patient race on the use of diagnostic imaging in united states emergency departments: data from the national hospital ambulatory medical care survey.  _BMC Health Services Research_ , 20(1):1–10, 2020\.
  * Shang et al. [2019] J. Shang, T. Ma, C. Xiao, and J. Sun.  Pre-training of graph augmented transformers for medication recommendation.  _arXiv preprint arXiv:1906.00346_ , 2019.
  * Sheng et al. [2019] E. Sheng, K.-W. Chang, P. Natarajan, and N. Peng.  The woman worked as a babysitter: On biases in language generation.  _arXiv preprint arXiv:1909.01326_ , 2019.
  * Singhal et al. [2022] K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi, J. Wei, H. W. Chung, N. Scales, A. Tanwani, H. Cole-Lewis, S. Pfohl, et al.  Large language models encode clinical knowledge.  _arXiv preprint arXiv:2212.13138_ , 2022.
  * Sirrianni et al. [2022] J. Sirrianni, E. Sezgin, D. Claman, and S. L. Linwood.  Medical text prediction and suggestion using generative pretrained transformer models with dental medical notes.  _Methods of Information in Medicine_ , 61(05/06):195–200, 2022.
  * Stephenson [2001] J. Stephenson.  Racial barriers may hamper diagnosis, care of patients with alzheimer disease.  _JAMA_ , 286(7):779–780, 2001.
  * Sun et al. [2022] M. Sun, T. Oliwa, M. E. Peek, and E. L. Tung.  Negative patient descriptors: Documenting racial bias in the electronic health record: Study examines racial bias in the patient descriptors used in the electronic health record.  _Health Affairs_ , 41(2):203–211, 2022.
  * Thoppilan et al. [2022] R. Thoppilan, D. De Freitas, J. Hall, N. Shazeer, A. Kulshreshtha, H.-T. Cheng, A. Jin, T. Bos, L. Baker, Y. Du, et al.  Lamda: Language models for dialog applications.  _arXiv preprint arXiv:2201.08239_ , 2022.
  * Tschandl et al. [2019] P. Tschandl, C. Rosendahl, B. N. Akay, G. Argenziano, A. Blum, R. P. Braun, H. Cabo, J.-Y. Gourhant, J. Kreusch, A. Lallas, et al.  Expert-level diagnosis of nonpigmented skin cancer by combined convolutional neural networks.  _JAMA dermatology_ , 155(1):58–65, 2019.
  * Uthoff et al. [2019] J. Uthoff, P. Nagpal, R. Sanchez, T. J. Gross, C. Lee, and J. C. Sieren.  Differentiation of non-small cell lung cancer and histoplasmosis pulmonary nodules: insights from radiomics model performance compared with clinician observers.  _Translational lung cancer research_ , 8(6):979, 2019.
  * van Wezel et al. [2021] M. M. van Wezel, E. A. Croes, and M. L. Antheunis.  “i’m here for you”: Can social chatbots truly support their users? a literature review.  In _Chatbot Research and Design: 4th International Workshop, CONVERSATIONS 2020, Virtual Event, November 23–24, 2020, Revised Selected Papers 4_ , pages 96–113. Springer, 2021.
  * Wang et al. [2021] L. Wang, M. I. Mujib, J. Williams, G. Demiris, and J. Huh-Yoo.  An evaluation of generative pre-training model-based therapy chatbot for caregivers.  _arXiv preprint arXiv:2107.13115_ , 2021.
  * Ware et al. [2020] O. R. Ware, J. E. Dawson, M. M. Shinohara, and S. C. Taylor.  Racial limitations of fitzpatrick skin type.  _Cutis_ , 105(2):77–80, 2020.
  * Weidinger et al. [2021] L. Weidinger, J. Mellor, M. Rauh, C. Griffin, J. Uesato, P.-S. Huang, M. Cheng, M. Glaese, B. Balle, A. Kasirzadeh, et al.  Ethical and social risks of harm from language models.  _arXiv preprint arXiv:2112.04359_ , 2021.
  * Weiss et al. [2016] K. Weiss, T. M. Khoshgoftaar, and D. Wang.  A survey of transfer learning.  _Journal of Big data_ , 3(1):1–40, 2016.
  * Willemink et al. [2020] M. J. Willemink, W. A. Koszek, C. Hardell, J. Wu, D. Fleischmann, H. Harvey, L. R. Folio, R. M. Summers, D. L. Rubin, and M. P. Lungren.  Preparing medical imaging data for machine learning.  _Radiology_ , 295(1):4–15, 2020.
  * Zhang et al. [2022] S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, et al.  Opt: Open pre-trained transformer language models.  _arXiv preprint arXiv:2205.01068_ , 2022.
  * Zhao et al. [2014] W. Zhao, P. T. Katzmarzyk, R. Horswell, Y. Wang, J. Johnson, and G. Hu.  Sex differences in the risk of stroke and hba 1c among diabetic patients.  _Diabetologia_ , 57:918–926, 2014.
  * Zhou et al. [2021] K. Zhou, K. Ethayarajh, and D. Jurafsky.  Frequency-based distortions in contextualized word embeddings.  _arXiv preprint arXiv:2104.08465_ , 2021.


# High Speed Imaging of Spectral-Temporal Correlations in Hong-Ou-Mandel
Interference

Yingwen Zhang  1 Duncan England  1,* Andrei Nomerotski  2 and Benjamin
Sussman1,3  1National Research Council of Canada, 100 Sussex Drive, Ottawa ON
Canada, K1A0R6
2Physics Department, Brookhaven National Laboratory, Upton, NY 11973, USA
3Department of Physics, University of Ottawa, Ottawa, Ontario, K1N 6N5, Canada
<EMAIL_ADDRESS>
###### Abstract

In this work we demonstrate spectral-temporal correlation measurements of the
Hong-Ou-Mandel (HOM) interference effect with the use of a 3D (2 spatial + 1
time dimension) camera based spectrometer. This setup allows us to take,
within seconds, spectral temporal correlation measurements on entangled photon
sources with sub-nanometer spectral resolution and nanosecond timing
resolution. Through post processing, we can observe the HOM behaviour for any
number of spectral filters of any shape and width at any wavelength over the
observable spectral range. Our setup also offers great versatility in that it
is capable of operating at a wide spectral range from the visible to the near
infrared and does not require a pulsed pump laser for timing purposes. This
work offers the ability to gain large amounts of spectral and temporal
information from a HOM interferometer quickly and efficiently and will be a
very useful tool for many quantum technology applications and fundamental
quantum optics research.

††journal: osajournal††articletype: Research Article

##  1 Introduction

The interference of two photons on a beamsplitter (BS), the Hong-Ou-Mandel
(HOM) effect [1], is one of the most defining features of quantum mechanics
and lies at the heart of many technology applications requiring multi-photon
interference including quantum communications, quantum computing, quantum
state engineering and precision measurements [2, 3, 4, 5, 6, 7, 8, 9, 10]. The
characteristic signature of HOM interference is a ‘dip’ in two-fold
coincidences in the output ports of the beamsplitter. It has become synonymous
with HOM interference, but in reality this dip is only a single-mode
representation of a far more complex multi-mode effect. Recent demonstrations
of the HOM effect in spatial-temporal [11], and spectral-temporal [12, 13, 14,
15] modes have revealed the complex multi-dimensional nature of HOM
interference.

Here we use a two-photon spectrometer based on a novel photon-counting camera
[16, 17, 18] to spectrally resolve the HOM effect with high spectro-temporal
resolution. Each output port of the beasmplitter (BS) is spectrally binned
into 256 pixels and coincident detection is performed in all $256\times
256=65536$ joint spectral modes simultaneously as the arrival time between the
photons is scanned. In this way, a full spectro-temporal characterization of
the HOM effect can be performed within just a few minutes. Conventional single
photon cameras, such as EMCCD and ICCD cameras, limited by their frame rate,
will require hours if not days to perform the same measurement and are also
prone to stability issues within the long data taking period. It is also well
known that the shape and width of the HOM interference dip is closely related
to the photon spectrum and type of spectral filter used to perform the
experiment [19, 20, 21]. Conventionally, investigating the spectral response
of HOM interference may require the use of many different spectral filters or
the use of pulse shapers. However, with our system, all such spectral
information can be obtained in just one data taking run. Through post-
processing, we have the freedom to artificially add any number of spectral
filters with any wavelength, bandwidth and shape so long it is within the
spectral range of the spectrometer. Compared to using a fiber-based
spectrometer [12, 13], or a chirped fibre bragg grating [22, 14], our camera
based spectrometer can operate in a much wider spectrum from around 200 nm to
900 nm depending on the spectral response of the image intensifier used. We
are also not limited to analysing photon pairs generated from a pulsed laser
pump source, often required for timing purposes, thus giving our system more
versatility.

##  2 Theory

The spectral correlation between the the photon pairs generated from SPDC can
be displayed in a joint spectral intensity (JSI) plot. Theoretically this can
be determined by multiplying the phase-matching function of the crystal and
the pump envelope function. The phase matching function for a ppKTP crystal of
length $L$ with Type-0 phase matching is given by

$h(\omega_{1},\omega_{2})=\text{sinc}[L\Delta k(\omega_{1},\omega_{2})]$ (1)

with

$\Delta k(\omega_{1},\omega_{2})=\frac{1}{2\pi
c}\left[n_{e}(\omega_{p})\omega_{p}-n_{e}(\omega_{1})\omega_{1}-n_{e}(\omega_{2})\omega_{2}\right]-\frac{1}{\Lambda_{0}},$
(2)

where $\omega_{p}$, $\omega_{1}$ and $\omega_{2}$ are the frequency of the
pump and photon pairs respectively. $n_{e}(\omega)$ is the extraordinary
refractive index of the crystal, $\Lambda_{0}$ is the grating period of the
crystal and $c$ the speed of light in vacuum.

Assuming a Gaussian shaped pump, the pump envelope function is given by

$\alpha(\omega_{1},\omega_{2})=\exp\left[-\frac{\left(\omega_{p}-\omega_{1}-\omega_{2}\right)^{2}}{2\sigma_{p}^{2}}\right],$
(3)

with $\sigma_{p}$ the pump bandwidth.

Finally the JSI is given by

$\Phi(\omega_{1},\omega_{2})=|\alpha(\omega_{1},\omega_{2})h(\omega_{1},\omega_{2})|^{2}.$
(4)

Now with an expression for the JSI we can use it to calculate the shape of the
HOM dip. As shown in [1, 19], the intensity correlation function for the HOM
interference with multi-spectral modes is given by

$G^{(2)}(t_{1},t_{2},\Delta z)\propto\left|\int
d\omega_{1}d\omega_{2}\Phi(\omega_{1},\omega_{2})\left(Te^{i\omega_{1}\Delta
z/c}-Re^{i\omega_{2}\Delta
z/c}\right)e^{-it_{1}\omega_{1}-it_{2}\omega_{2}}\right|^{2},$ (5)

with $\Delta z$ the path length difference, $T$ and $R$ the transmissivity and
reflectivity of the BS.

Since the detector timing resolution is much longer than the coherence time of
the SPDC photons, which is approximately the inverse of the spectral filter
bandwidth, the two photon coincidence count is then proportional to the time
average of $G^{(2)}(t_{1},t_{2})$

$\displaystyle N_{c}(\Delta z)$
$\displaystyle\propto\int_{\infty}^{\infty}dt_{1}dt_{2}G^{(2)}(t_{1},t_{2},\Delta
z)$ $\displaystyle\propto\left|\int
d\omega_{1}d\omega_{2}\Phi(\omega_{1},\omega_{2})\left(Te^{i\omega_{1}\Delta
z/c}-Re^{i\omega_{2}\Delta z/c}\right)\right|^{2}.$ (6)

A useful observation here is the absolute square of the integrand in Eq. 6,

$\left|\Phi(\omega_{1},\omega_{2})\left(Te^{i\omega_{1}\Delta
z/c}-Re^{i\omega_{2}\Delta z/c}\right)\right|^{2},$ (7)

will be the JSI measured between the two output ports of the BS in HOM
interference. Oscillations in the intensity will be observed with a frequency
of $2\Delta z/c$.

When spectral filters with spectral shape of $f(\omega)$ are placed in the
system, a small modification is required in the JSI,
$\Phi(\omega_{1},\omega_{2})\rightarrow\Phi^{\prime}(\omega_{1},\omega_{2})$,
where

$\Phi^{\prime}(\omega_{1},\omega_{2})=f(\omega_{1})f(\omega_{2})\Phi(\omega_{1},\omega_{2}).$
(8)

##  3 Experimental Setup

Figure 1: (a) Time and wavelength correlated photon pairs are generated
through SPDC by a 405.5 nm CW laser pumping a nonlinear ppKTP crystal. A long-
pass (LP) spectral filter is used to filter out the pump laser. The generated
signal and idler photons are then sent on different paths and undergo HOM
interference at a 50:50 BS before being coupled into single-mode (SM) optical
fibers. A motorized translational stage is used to control the arrival time
difference between the two photons at the BS. The photons are vertically
displaced upon exiting their respective fibres and they are diffracted off a
diffraction grating which decomposes the photons into their spectral
compositions. The diffracted photons are then focused by a lens onto the 3D
camera. (b) Image of the twin beam spectrum captured on the camera.

The experimental setup is shown in Fig. 1(a). A 200 mW, 405.5 nm continuous
wave (CW) laser is used to pump a 1 mm long Type-0 periodically poled
potassium titanyl phosphate (ppKTP) crystal to generate photon pairs through
the process of SPDC. A 10 cm focal length lens is placed after the crystal to
collimate the SPDC beam. The photon pairs are then split into separate paths
using a D-shaped mirror and are then sent into a 50:50 BS (actual T:R ratio is
57:43) to undergo HOM interference. The difference in arrival time between the
photon pairs at the BS is adjusted through a motorized delay line constructed
in one of the paths. Upon exiting the BS, the photons are coupled into single
mode optical fibers which are then fed into the customized spectrometer [23,
24, 25].

Within the spectrometer, the exit of the fibers are set at a slightly
different height with respect to each other and the two beams are diffracted
off a diffraction grating with the first diffraction order focused onto the 3D
camera (TPX3CAM [26]) using a 2 inch diameter, 10 cm focal length lens. The
two beams are made to not propagate perfectly in parallel so to prevent the
lens from focusing both spectra onto the same strip on the camera. The timing
and spectral information of the photons are registered by the 3D camera from
which we are able to deduce the temporal and spectral correlations between the
photons. Single photon sensitivity of the 3D camera is provided by the
attached image intensifier which has a quantum efficiency of approximately 20%
in the spectral range of 500-850 nm [27]. A raw image of the twin beam
spectrum captured on the camera is seen in Fig. 1(b).

As a single photon is converted into a flash of light by the image
intensifier, a cluster of pixels will often be illuminated on the camera. Such
a cluster has to be regrouped into a single event through a detection and
centroiding algorithm [16]. After this correction, a spectral resolution of
approximately 0.7 nm is obtained over a spectral range of 770-854 nm. Due to
the signal strength (photon flux) dependence of the camera’s discrimination
threshold crossover time [16, 28], a timing correction must also be performed,
after which a timing resolution of approximately 6 ns is achieved.

##  4 Results

Figure 2: (a) A typical JSI measured after the BS at a path length difference
of $\Delta z=0.048$ mm. (b) Cross-section along the bright central band of the
experimentally measured JSI as a function of the path length difference
$\Delta z$ with $\lambda_{1}-\lambda_{2}=0$ the degenerate wavelength. The
cross-section of (a) highlighted by the red line. The corresponding
theoretical JSI are shown in (c) and (d). A total of 151 JSI slices were taken
in (b) between $\Delta z=-0.3$ mm and $0.3$ mm, the data acquisition time is
10 seconds with a total of $\sim 10^{4}$ coincidence events for each JSI
slice. Note that the coincidence counts here are not the total coincidence
$N_{c}(\Delta z)$ given by Eq. 6 but the coincidence for each wavelength pair
given by Eq. 7.

Experimentally, through looking for time correlated events between the two
spectral bands seen in Fig. 1(b) and then by recording the two corresponding
x-pixel/wavelength of the correlated photons, a JSI plot can be made. A
typical JSI experimentally measured after the BS at a path length difference
of $\Delta z=0.048$ mm is shown in Fig. 2(a). The cross-section along the
bright central band of the experimentally measured JSI as a function of
$\Delta z$ is shown in Fig. 2(b). We see fast oscillations in the coincidence
rate when away from $\Delta z=0$ and no coincidences at $\Delta z=0$. The
corresponding theoretical JSI, calculated from the integrand of Eq. 7, are
shown in Fig. 2(c) and (d). Very Good agreement is seen between the
experimental results and theory.

In order to calculate the theoretical JSI, we first calculated the phase
matching function (Eq. 1 and 2) with $\lambda_{p}=2\pi c/\omega_{p}=405.5$ nm,
$\Lambda_{0}=3.425$ $\mu$m and $L=1$ mm. $n_{e}(\omega)$ is obtained using the
Sellmeier equation given in [29] for a temperature of $24^{\circ}\text{C}$.
The pump envelope function (Eq. 4) is determined using a pump bandwidth
$\sigma_{p}$ of 1 nm.

Figure 3: (a) The HOM interference dip for top-hat shaped spectral filters
centered on the degenerate wavelength of 811 nm with a bandwidth of 40 nm
(black), 7 nm (blue) and 3 nm (red). Circles are experimental data and the
corresponding solid line with the same color is the predicted shape based on
theoretical calculation. (b)-(d) highlights the corresponding region in the
full JSI for the various filter bandwidth. The displayed JSI is taken at path
length difference of $\Delta z=0.3$ mm.

Since the measured JSI in fig. 2 contains all the coincidence events across
the full measurable spectrum of the spectrometer, one can through post-
processing, place virtual spectral filters over the measured JSI to look at
the changes in coincidence events in any spectral region of interest. In Fig.
3, we show the HOM dip for top-hat shaped spectral filters of various
bandwidth (3, 7 and 40 nm) centered on the degenerate wavelength of 811 nm
extracted from the HOM dip scan data taken with no physical spectral filters
used. The shape of the HOM dip between theoretical calculation (through Eqs. 6
and 8 using a BS T:R ratio of 57:43) and experimental data is shown in Fig.
3(a). Fig. 3(c)-(d) highlights the corresponding region in the full JSI, taken
at $\Delta z=0.3$ mm, for the various filter bandwidth. The experimental dip
did not fully reach the theoretical minimum is due to imperfections in the
optics alignment.

Figure 4: HOM interference dips for (a) a Gaussian shaped 811/5 nm spectral
filter, (b) an absolute value Sinc function shaped 811/5 nm spectral filter,
(c) dual band top-hat shaped filters centred on 805/2 nm and 817/2 nm and (d)
quad band top-hat shaped filters centred on 799/1 nm, 801/1 nm 821.3/1 nm and
823.4/1 nm. Circles are experimental data and the solid line is the predicted
shape based on theoretical calculation. The corresponding shape of the
spectral filters are shown in the insets.

Of course, one is also free to choose the shape and number of spectral filters
to be placed and this is demonstrated in Fig. 4. Fig. 4(a) and (b) shows the
HOM dip scan of a Gaussian and an absolute value Sinc function shaped 811/5 nm
spectral filter, where a Gaussian and triangular shaped HOM dip are observed.
A dual band top-hat filter centered at 805/2nm and 817.1/2 nm is used in Fig.
4(c) where oscillations are seen in the HOM dip. Finally, in Fig. 4(d), a quad
band top-hat shaped shaped filters centred on 799/1 nm, 801/1 nm 821.3/1 nm
and 823.4/1 nm is shown, in which the combination of narrow filters and small
central wavelength difference results in a beating pattern in the HOM dip
oscillation.

It is also possible to observe photon bunching events in HOM from the same
data set by looking for coincidences in the spectrum of the same beam instead
of between the two beams[30, 18]. However, due to the requirement to correct
for clustering of events caused by the image intensifier, as discussed in the
previous section, coincidence events that occur approximately in a 10 pixel
radius around the degenerate wavelength of 811 nm will all be regrouped into a
single event. This results in the inability to obtain an accurate HOM ‘peak’
for single band spectral filters centered around the degenerate wavelength of
811 nm. However, for non-degenerate wavelengths, as seen in Fig. 5, accurate
HOM interference shape for bunching events can still be obtained.

Figure 5: The HOM interference for bunching (blue) and anti-bunching (red)
events with dual band top-hat shaped filters centred on 805/2 nm and 817/2 nm.
Circles are experimental data and the corresponding solid line with the same
color is the predicted shape based on theoretical calculation. The bunching
events shown are the sum of all bunching events from both output channels.

##  5 Discussion

In this work we demonstrated measurement of the spectral and temporal
correlations in HOM interference using a camera based spectrometer. The
spectral response to HOM interference for the full operating spectrum of the
spectrometer, between 770-854 nm, is obtained with a spectral resolution of
0.7 nm. From this, the HOM interference behaviour at any specific spectral
region/s within the measured spectrum can be extracted. Thus, with this
spectrometer, full spectral and temporal correlation data, with sub-nanometer
spectral and nanosecond temporal resolution, can be obtained within seconds
and a full HOM dip/peak scan can be performed in minutes. The system is also
versatile in that it can be used with SPDC sources pumped by any type of laser
and can work in any wavelength that an image intensifier is sensitive in. All
these technical improvements combined, especially in speed, opens up a whole
new avenue on what is practicable in quantum technology applications and
experiments. Some examples include spectral multiplexing in quantum repeaters,
obtaining spectral information of targets through a single shot measurement in
quantum sensing and swiftly characterizing spectral-temporal properties of
single photon sources.

Improvements to the spectral resolution can be achieved with a better designed
spectrometer and through predicting the initial position of the photon through
a centroiding algorithm and subdividing each camera pixel, thus giving a
higher spatial resolution to the camera than the physical number of pixels
[31]. The temporal resolution can also be improved to sub-nanoseconds with the
use of a trigger signal from the multi-channel plates of the image intensifier
[32] or, as proposed in the same work, using of a streaking tube in place of
the image intensifier. The measurement speed is only limited by the photon
pair production rate and the overall detection efficiency of the system, a
brighter source and more efficient optical elements (mostly the diffraction
grating) will further improve the overall measurement speed.

## Acknowledgements

The authors are grateful to Philip Bustard, Frédéric Bouchard, Khabat Heshami,
Rune Lausten, Denis Guay, and Doug Moffatt for technical support and
stimulating discussion. We acknowledge support from Defence Research and
Development Canada (DRDC).

## References

  * [1] C. K. Hong, Z. Y. Ou, and L. Mandel, “Measurement of subpicosecond time intervals between two photons by interference,” Phys. Rev. Lett. 59, 2044–2046 (1987).
  * [2] N. Sangouard, C. Simon, H. de Riedmatten, and N. Gisin, “Quantum repeaters based on atomic ensembles and linear optics,” Rev. Mod. Phys. 83, 33–80 (2011).
  * [3] A. Acín, N. Brunner, N. Gisin, S. Massar, S. Pironio, and V. Scarani, “Device-independent security of quantum cryptography against collective attacks,” Phys. Rev. Lett. 98, 230501 (2007).
  * [4] E. Knill, R. Laflamme, and G. J. Milburn, “A scheme for efficient quantum computation with linear optics,” Nature 409, 46–52 (2001).
  * [5] Y. Zhang, F. S. Roux, T. Konrad, M. Agnew, J. Leach, and A. Forbes, “Engineering two-photon high-dimensional states through quantum interference,” Science Advances 2 (2016).
  * [6] T.-M. Zhao, H. Zhang, J. Yang, Z.-R. Sang, X. Jiang, X.-H. Bao, and J.-W. Pan, “Entangling different-color photons via time-resolved measurement and active feed forward,” Phys. Rev. Lett. 112, 103602 (2014).
  * [7] C. Vitelli, N. Spagnolo, L. Aparo, F. Sciarrino, E. Santamato, and L. Marrucci, “Joining the quantum state of two photons into one,” Nature Photonics 7, 521–526 (2013).
  * [8] A. Lyons, T. Roger, N. Westerberg, S. Vezzoli, C. Maitland, J. Leach, M. J. Padgett, and D. Faccio, “How fast is a twisted photon?” Optica 5, 682–686 (2018).
  * [9] A. Lyons, G. C. Knee, E. Bolduc, T. Roger, J. Leach, E. M. Gauger, and D. Faccio, “Attosecond-resolution hong-ou-mandel interferometry,” Science Advances 4 (2018).
  * [10] F. Bouchard, A. Sit, Y. Zhang, R. Fickler, F. M. Miatto, Y. Yao, F. Sciarrino, and E. Karimi, “Two-photon interference: the hong–ou–mandel effect,” Reports on Progress in Physics 84, 012402 (2021).
  * [11] F. Devaux, A. Mosset, P.-A. Moreau, and E. Lantz, “Imaging spatiotemporal hong-ou-mandel interference of biphoton states of extremely high schmidt number,” Phys. Rev. X 10, 031031 (2020).
  * [12] T. Gerrits, F. Marsili, V. B. Verma, L. K. Shalm, M. Shaw, R. P. Mirin, and S. W. Nam, “Spectral correlation measurements at the hong-ou-mandel interference dip,” Phys. Rev. A 91, 013830 (2015).
  * [13] R.-B. Jin, T. Gerrits, M. Fujiwara, R. Wakabayashi, T. Yamashita, S. Miki, H. Terai, R. Shimizu, M. Takeoka, and M. Sasaki, “Spectrally resolved hong-ou-mandel interference between independent photon sources,” Opt. Express 23, 28836–28848 (2015).
  * [14] V. Thiel, A. O. C. Davis, K. Sun, P. D’Ornellas, X.-M. Jin, and B. J. Smith, “Single-photon characterization by two-photon spectral interferometry,” Opt. Express 28, 19315–19324 (2020).
  * [15] Y. Chen, S. Ecker, L. Chen, F. Steinlechner, M. Huber, and R. Ursin, “Temporal distinguishability in hong-ou-mandel interference: Generation and characterization of high-dimensional frequency entanglement,” (2020).
  * [16] C. Ianzano, P. Svihra, M. Flament, A. Hardy, G. Cui, A. Nomerotski, and E. Figueroa, “Fast camera spatial characterization of photonic polarization entanglement,” Scientific Reports 10, 6181 (2020).
  * [17] A. Nomerotski, “Imaging and time stamping of photons with nanosecond resolution in timepix based optical cameras,” Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 937, 26 – 30 (2019).
  * [18] A. Nomerotski, M. Keach, P. Stankus, P. Svihra, and S. Vintskevich, “Counting of hong-ou-mandel bunched optical photons using a fast pixel camera,” Sensors 20, 3475 (2020).
  * [19] Z. Y. Ou, _Multi-Photon Quantum Interference_ (Springer Science & Business Media, LLC, 233 Spring Street, New York, NY 10013, USA, 2007).
  * [20] C. Liu, J. F. Chen, S. Zhang, S. Zhou, Y.-H. Kim, M. M. T. Loy, G. K. L. Wong, and S. Du, “Two-photon interferences with degenerate and nondegenerate paired photons,” Phys. Rev. A 85, 021803 (2012).
  * [21] N. B. Lingaraju, H.-H. Lu, S. Seshadri, P. Imany, D. E. Leaird, J. M. Lukens, and A. M. Weiner, “Quantum frequency combs and hong ou mandel interferometry: the role of spectral phase coherence,” Opt. Express 27, 38683–38697 (2019).
  * [22] A. O. C. Davis, P. M. Saulnier, M. Karpiński, and B. J. Smith, “Pulsed single-photon spectrometer by frequency-to-time mapping using chirped fiber bragg gratings,” Opt. Express 25, 12804–12811 (2017).
  * [23] K. D. Johnsen2014, P. Kolenderski, C. Scarcella, M. Thibault, A. Tosi, and T. Jennewein, “Time and spectrum-resolving multiphoton correlator for 300–900 nm,” Journal of Applied Physics 116, 143101 (2014).
  * [24] Y. Zhang, D. England, A. Nomerotski, P. Svihra, S. Ferrante, P. Hockett, and B. Sussman, “Multidimensional quantum-enhanced target detection via spectrotemporal-correlation measurements,” Phys. Rev. A 101, 053808 (2020).
  * [25] P. Svihra, Y. Zhang, P. Hockett, S. Ferrante, B. Sussman, D. England, and A. Nomerotski, “Multivariate discrimination in quantum target detection,” Applied Physics Letters 117, 044001 (2020).
  * [26] “https://www.amscins.com/tpx3cam/,” .
  * [27] “https://www.photonis.com/products/crickettm,” .
  * [28] A. Zhao, M. van Beuzekom, B. Bouwens, D. Byelov, I. Chakaberia, C. Cheng, E. Maddox, A. Nomerotski, P. Svihra, J. Visser, V. Vrba, and T. Weinacht, “Coincidence velocity map imaging using tpx3cam, a time stamping optical camera with 1.5 ns timing resolution,” Review of Scientific Instruments 88, 113104 (2017).
  * [29] S. Manjooran, H. Zhao, I. T. Lima, and A. Major, “Phase-matching properties of ppktp, mgo:ppslt and mgo:ppcln for ultrafast optical parametric oscillation in the visible and near-infrared ranges with green pump,” Laser Physics 22, 1325–1330 (2012).
  * [30] G. D. Giuseppe, M. Atatüre, M. D. Shaw, A. V. Sergienko, B. E. A. Saleh, M. C. Teich, A. J. Miller, S. W. Nam, and J. Martinis, “Direct observation of photon pairs at a single output port of a beam-splitter interferometer,” Physical Review A 68 (2003).
  * [31] G. Kim, K. Park, K. Lim, J. Kim, and G. Cho, “Improving spatial resolution by predicting the initial position of charge-sharing effect in photon-counting detectors,” Journal of Instrumentation 15, C01034–C01034 (2020).
  * [32] A. Nomerotski, P. Stankus, A. Slosar, S. Vintskevich, S. Andrewski, G. Carini, D. Dolzhenko, D. England, E. Figueroa, S. Gera, J. Haupt, S. Herrmann, D. Katramatos, M. Keach, A. Parsells, O. Saira, J. Schiff, P. Svihra, T. Tsang, and Y. Zhang, “Quantum-assisted optical interferometers: Instrument requirements,” (2020).