11institutetext: Department of Computer Science, School of Science,
Loughborough University, UK 22institutetext: School of Design and Creating
Arts, Loughborough University, UK 33institutetext: Health Services Safety
Investigations Body (HSSIB), United Kingdom 33email<EMAIL_ADDRESS>

# Unveiling Disparities in Maternity Care: A Topic Modelling Approach to
Analysing Maternity Incident Investigation Reports

Georgina Cosma 11 Mohit Kumar Singh 11 Patrick Waterson 22 Gyuchan Thomas Jun
22 Jonathan Back 33

###### Abstract

This study applies Natural Language Processing techniques, including Latent
Dirichlet Allocation, to analyse anonymised maternity incident investigation
reports from the Healthcare Safety Investigation Branch. The reports underwent
preprocessing, annotation using the Safety Intelligence Research taxonomy, and
topic modelling to uncover prevalent topics and detect differences in
maternity care across ethnic groups. A combination of offline and online
methods was utilised to ensure data protection whilst enabling advanced
analysis, with offline processing for sensitive data and online processing for
non-sensitive data using the ‘Claude 3 Opus’ language model. Interactive topic
analysis and semantic network visualisation were employed to extract and
display thematic topics and visualise semantic relationships among keywords.
The analysis revealed disparities in care among different ethnic groups, with
distinct focus areas for the Black, Asian, and White British ethnic groups.
The study demonstrates the effectiveness of topic modelling and NLP techniques
in analysing maternity incident investigation reports and highlighting
disparities in care. The findings emphasise the crucial role of advanced data
analysis in improving maternity care quality and equity.

###### Keywords:

Topic modelling Maternity Care Healthcare safety

##  1 Introduction

Maternity hospital deaths have been a significant concern in the United
Kingdom, and the prevention of future deaths (PFD) reports have played a
crucial role in addressing this issue. The Healthcare Safety Investigation
Branch (HSIB) was an independent organisation established to investigate
patient safety incidents within the NHS in England. Established in 2017 and
funded by the Department of Health and Social Care, HSIB aimed to improve
patient safety through its investigations. In October 2023, HSIB transformed
into two organisations: the Maternity and Newborn Safety Investigations
(MNSI), which is hosted by the Care Quality Commission, and the Health
Services Safety Investigations Body (HSSIB), which is an independent statutory
body. HSIB’s maternity investigation programme focused on identifying systemic
factors contributing to adverse events in maternity care. The programme
investigated qualifying incidents referred by NHS trusts, such as intrapartum
stillbirths, early neonatal deaths, and severe brain injuries in term babies.
The HSIB Maternity Investigation Programme Year in Review 2022/23 [1] provides
an overview of the programme’s activities and findings from 1 April 2022 to 31
March 2023. During this period, HSIB received 1 070 referrals for maternity
incidents, of which 671 progressed to full investigations. The most common
type of incident investigated was term babies who were cooled or diagnosed
with brain injuries (52%), followed by intrapartum stillbirths (26%), early
neonatal deaths (13%), and maternal deaths (9%). The review highlights a
reduction in referrals for severe brain injury in babies over the past few
years, with a 13% decrease in 2022/23 compared to the previous year and a 19%
decline compared to 2019/20. The investigations also identified recurring
themes, such as issues with clinical assessment, guidance adherence, foetal
monitoring, escalation protocols, and communication among healthcare
professionals. One of the most high-profile cases that led to a PFD report was
the death of baby Harry Richford at East Kent Hospitals University NHS
Foundation Trust in 2017 [5]. The coroner’s report outlines numerous failures
and concerns that include the inadequate assessment and supervision of a locum
registrar, confusion among staff regarding guidelines and when to escalate
care, insufficient neonatal resuscitation training, poor record-keeping, and
failure to properly report the death. The coroner made 19 recommendations for
the Trust and national bodies to address these problems, such as reviewing
locum recruitment, clarifying policies, improving training, and auditing
practices, to prevent future deaths.
The Ockenden Review, an independent inquiry into maternity services at the
Shrewsbury and Telford Hospital NHS Trust, was initiated due to concerns
raised by the families of babies who died or were harmed in the maternity
units of the Trust [2]. The Ockenden Report scrutinises maternity services at
Shrewsbury and Telford Hospital NHS Trust from 2000 to 2019, covering 1 486
families and 1 592 clinical incidents. The report highlights a culture of poor
communication with affected families and insufficient adherence to clinical
guidelines, underlining an urgent need for improved leadership, transparency,
and a comprehensive maternity workforce plan to prevent future failings.
The Royal College of Obstetricians and Gynaecologists (RCOG) has also taken
steps to address the issues highlighted in PFD reports. The “Each Baby Counts:
2020 Final Progress Report” [3] by the Royal College of Obstetricians and
Gynaecologists (RCOG) presents a critical evaluation of maternity care
outcomes in the UK, focusing on 2018 data involving 1 145 cases of
stillbirths, neonatal deaths, and severe brain injuries during term labour. A
pivotal finding was that in 74% of cases, different care might have led to
different outcomes, suggesting systemic deficiencies in clinical governance,
risk assessment, and neonatal care. The report underscores the importance of
multidisciplinary reviews and enhanced parental involvement, with 70% of
parents invited to contribute to the review process. Despite the challenges in
achieving its ambitious goal to halve these tragic incidents by 2020, the
initiative significantly raised maternity safety awareness and set a
foundation for future improvements. The Each Baby Counts programme, through
its comprehensive analysis and recommendations, leaves a lasting legacy,
advocating for sustained efforts to address disparities and improve maternity
care services across the UK.
This paper utilises a range of Natural Language Processing (NLP) techniques to
analyse 188 maternity incident reports provided by HSIB to identify
disparities in care across different ethnic groups, underscoring the critical
need for reforms to enhance maternity care quality and equity.

##  2 Methodology

###  2.1 Dataset of maternity incident investigation reports

The Healthcare Safety Investigation Branch (HSIB) provided a random set of 188
anonymised investigation reports describing adverse maternity incidents. The
reports were written between 2019 to 2022. The number of reports for each year
is as follows: 4 reports in 2019, 115 reports in 2020, 42 reports in 2021, and
27 reports in 2022. Ethnicity was provided for 76 reports. The reports were
processed, and the output was a text file (in CSV format) containing File IDs,
Sentence IDs, sentence, and concepts from the Safety Intelligence ResearCH
(SIRch) taxonomy. During the processing phase, the files underwent cleaning,
involving the selection of specific sentences from each report. The sentence
selection process focused on identifying sentences that contained negative
connotations, references to physical characteristics, and mentions of
medication names related to dispensing medications. The selected sentences
were annotated using the SIRch taxonomy and compiled into a CSV file, which is
utilised for clustering and summarisation tasks. This concept annotation task
was carried out using our I-SIRch tool, and the process is described in [4].
Table 1 shows the list of Concepts with the number of sentences per concept
shown in brackets.

Table 1: ID, concept and number of sentences per concept. Total sentences: 3760. ID |  Concept | Index |  Label
---|---|---|---
1 |  Acuity (54) | 2 |  Antenatal (69)
3 |  Assessment, investigation, testing, screening (381) | 4 |  COVID (142)
5 |  Care Planning (132) | 6 |  Communication factor (477)
7 |  Decision error (169) | 8 |  Dispensing, administering (62)
9 |  Documentation (168) | 10 |  Escalation/referral factor (158)
11 |  Guidance factor (42) | 12 |  Interpretation (197)
13 |  Language barrier (30) | 14 |  Local guidance (88)
15 |  Monitoring (118) | 16 |  National and local guidance (80)
17 |  National guidance (169) | 18 |  Obstetric review (147)
19 |  Physical characteristics (320) | 20 |  Physical layout and Environment (35)
21 |  Psychological characteristics (54) | 22 |  Risk assessment (94)
23 |  Situation awareness (77) | 24 |  Slip or lapse (188)
25 |  Teamworking (530) | 26 |  Technologies and Tools-issues (112)
27 |  Training and education (40) |  |

###  2.2 Topic modelling and semantic network visualisation methods

####  2.2.1 Textual data analysis and topic modeling.

The methods proposed in this study focus on extracting and analysing topics
from a preprocessed dataset using Natural Language Processing (NLP)
techniques. The primary goal is to identify unique topics within the dataset
based on different combinations of concepts and ethnic groups. The process is
provided in Algorithm (1). All methods discussed in Section 2 are offline.
Topic Extraction: To extract topics from the dataset, the extract_topics_lda
function is defined. This function takes the DataFrame, a specific concept,
ethnicity, and the desired number of topics as input parameters. It subsets
the dataset based on the provided concept and ethnicity values, allowing for
the extraction of topics specific to particular combinations. TF-IDF
Vectorisation: The text data in the ‘Sentence’ column of the subset is
transformed into a numerical representation using the Term Frequency-Inverse
Document Frequency (TF-IDF) vectorisation technique. The resulting TF-IDF
matrix is then normalised using the L2 norm to eliminate discrepancies caused
by document size differences in the TF-IDF matrix. Latent Dirichlet Allocation
(LDA): The LDA model is applied to the normalised TF-IDF matrix, specifying
the desired number of topics. LDA is a probabilistic topic modelling algorithm
that discovers latent topics within a collection of documents. It assigns each
sentence to a mixture of topics and represents each topic as a distribution
over words.
Top Words Retrieval: After applying the LDA model, the top words associated
with each topic are retrieved. These top words provide a summarised
representation of the main topics discussed within each topic. Duplicate Topic
Removal: To remove duplicate topics the remove_duplicates function is defined.
It creates a new list containing only the unique topics from the input list,
discarding any duplicates. Topic Storage: The extracted topics, along with
their corresponding concept, ethnicity values, and sentence count are stored
in a dictionary. DataFrame Creation and Export: A new DataFrame is created
from the dictionary of topics. Each row in the DataFrame represents a unique
topic, along with its associated concept, ethnicity, topic number, keywords
and sentence count. The resulting DataFrame is then exported to a file
(extract shown in Table 2) for further analysis and visualisation.

Algorithm 1 Textual data analysis and topic modeling

1:Import data manipulation, text processing, and topic modeling libraries

2:Download NLTK resources including stopwords, tokenizers, and POS taggers

3:Load the preprocessed dataset from a csv file

4:procedure CleanText(text)

5: Remove non-ASCII characters from text

6: Tokenize text into words

7: Remove alphanumeric words, keeping alphabetic and numeric

8: Convert words to lowercase

9: Remove stopwords and tokens less than two characters

10: Perform POS tagging, keeping nouns and verbs

11: return cleaned text

12:end procedure

13:Apply CleanText to ‘Sentence’ column

14:procedure ExtractTopicsLDA(data, concept, ethnicity)

15: Subset data based on concept and ethnicity

16: if subset is empty then

17: return notification of no data

18: end if

19: Vectorize sentences using TF-IDF

20: Normalize TF-IDF vectors

21: Fit LDA model to normalized TF-IDF matrix

22: Extract top words for each topic

23: return topics

24:end procedure

25:Remove duplicate topics from list

26:Create a dictionary for unique topics per concept and ethnicity

27:for each unique concept and ethnicity combination do

28: Extract topics using ExtractTopicsLDA

29: Remove duplicates and store in dictionary

30: Count the number of sentences per topic

31:end for

32:Convert topics dictionary to list for DataFrame

33:Create DataFrame from list

34:Export DataFrame to a csv file for analysis or presentation

####  2.2.2 Interactive topic analysis and semantic network visualisation.

This section introduces computational methods designed to: extract and display
thematic topics from a dataset, and to visualise the semantic relationships
among the keywords of each topic. This dual approach is described as follows.
Visualise Topic Relationships: Complementing the thematic extraction, network
diagrams are generated to visualise the semantic relationships among the
keywords of each topic. These diagrams employ cosine similarity to map out the
connections between words, where nodes signify individual words, and edges
reflect the degree of semantic similarity. Such visualisations provide a
graphical representation of how topics are compositionally structured,
illustrating the semantic proximity and interconnectedness of topic keywords.
Interactivity and User Engagement: Users can specify the concept, ethnicity,
number of topics, and similarity threshold, among other parameters, thereby
facilitating a highly customised analysis that can adapt to varied research
needs. Algorithm (2) illustrates the process for constructing topic analysis
and network visualisation. The calculate_similarity function computes the
cosine similarity between terms using the TF-IDF matrix. It takes the entire
dataset or a subset and calculates the pairwise cosine similarity between all
terms. The create_network_diagram function constructs the network
visualisation. It extracts the top words for each topic and calculates the
cosine similarity between their TF-IDF vectors. A new graph (G) is
initialised, and for each pair of top words, an edge is added between them if
their cosine similarity exceeds a specified threshold. The layout for the
network visualisation is calculated, and the graph is drawn using matplotlib
with nodes, edges, and labels. The analysis and visualisation are executed by
calling the extract_topics_lda function to obtain the topics and their top
words. For each topic, the create_network_diagram function is called with the
topic’s top words, and the resulting network diagram is visualised.

Algorithm 2 Construct network diagrams from topics

1:procedure CalculateSimilarity($TFIDFMatrix$)

2: $SimilarityMatrix\leftarrow$ CosineSimilarity($TFIDFMatrix$)

3: return $SimilarityMatrix$

4:end procedure

5:procedure CreateNetworkDiagram($Topics$, $SimilarityMatrix$, $Threshold$)

6: for each $Topic$ in $Topics$ do

7: $TopWords\leftarrow$ ExtractTopWords($Topic$)

8: $G\leftarrow$ new Graph()

9: for each pair $(word1,word2)$ in $TopWords$ do

10: $similarity\leftarrow SimilarityMatrix[word1,word2]$

11: if $similarity>Threshold$ then

12: Add edge $(word1,word2)$ to $G$ with weight $similarity$

13: end if

14: end for

15: $Layout\leftarrow$ CalculateLayout($G$)

16: DrawGraph($G$, $Layout$)

17: end for

18:end procedure

19:procedure ExtractTopicsAndVisualize($DataFrame$, $Concept$, $Ethnicity$,
$NumTopics$)

20: $Topics,TFIDFMatrix\leftarrow$ ExtractTopicsLDA($DataFrame$, $Concept$,
$Ethnicity$, $NumTopics$)

21: $SimilarityMatrix\leftarrow$ CalculateSimilarity($TFIDFMatrix$)

22: CreateNetworkDiagram($Topics$, $SimilarityMatrix$, $Threshold$)

23:end procedure

##  3 Results

###  3.1 Analysis of concepts across each ethnic group

Table 2 provides an extract of the non-sensitive outputs when running the
proposed offline methods on the pre-processed dataset. The table organises LDA
analysis results into four columns: the first column details the SIRch concept
each topic pertains to, the second specifies the ethnic group associated, the
third column lists the topics identified by LDA, and the fourth shows the
number of sentences per topic for each ethnic-concept group, providing a
thorough overview of topic distribution and relevance within the LDA results.
Table 3 illustrates the distribution of five distinct topics across all ethnic
groups.
The complete version of Table 2 is provided in the project’s Github
repository111https://github.com/gcosma/I-SIRchpapers/. The topic analysis
results table (i.e. the full version of Table 2) was input into the ‘Claude 3
Opus’ online LLM for providing meaningful analysis and fast processing of non-
sensitive data. The results were human checked and no mistakes were found. Due
to space limitations, the discussion that follows will focus on the Asian,
Black, and White British ethnic groups.

Table 2: Topics in Care Planning (CP) across ethnic groups with the number of sentences per topic (S. Count). WB: White British. Conc. | Ethn. | Topic |  Keywords | S. Count
---|---|---|---|---
CP | WB | Topic 1 |  body, plans, occasions, period, mass, index, inclusion, inform, bmi, mothers | 3
CP | WB | Topic 2 |  baby, delivery, mother, birth, benefits, wishes, place, explore, plan, care | 4
CP | WB | Topic 3 |  change, gaps, situation, centile, team, weight, timing, labour, documentation, assessments | 3
CP | WB | Topic 4 |  pathway, opportunity, risk, pregnancy, staff, management, plan, care, discuss, meant | 8
CP | WB | Topic 5 |  monitoring, scenarios, clinicians, guidance, pathways, bp, evidence, triage, addition, advice | 2
CP | Asian | Topic 1 |  concerns, ctg, waters, cardiotocograph, times, reasons, possibility, plan, care, delays | 1
CP | Asian | Topic 2 |  concerns, ctg, waters, cardiotocograph, days, mother, ongoing, oversight, plan, care | 1
CP | Asian | Topic 3 |  ctg, waters, concerns, cardiotocograph, pathway, days, iol, possibility, care, consideration | 3
CP | Black | Topic 1 |  appointment, system, mother, treatment, place, plan, safety, care, antihistamines, assessment | 1
CP | Black | Topic 2 |  mother, risk, treatment, meant, tolerance, plan, care, antihistamines, assessment, test | 1
CP | Black | Topic 3 |  triggers, mother, anaphylaxis, risk, treatment, symptoms, delay, plan, care, antihistamines | 1
CP | Black | Topic 4 |  clinician, accumulation, mother, risk, treatment, risks, plan, care, lead, antihistamines | 1
CP | Black | Topic 5 |  mother, risk, treatment, meant, management, plan, delay, care, antihistamines, assessment | 2

Fig. 1 shows a network diagram for the ‘Care Planning’ concept for the Black
ethnic group. Similarity threshold was set to 0.50.

Figure 1: Concept: Care Planning, Ethnicity: Black. Similarity threshold: 0.50 Table 3: Sentence counts by ethnicity and topic. Up to 5 topics per group were generated. Duplicated topics were removed, hence some groups have $<5$ topics. Ethnicity | Topic 1 | Topic 2 | Topic 3 | Topic 4 | Topic 5 | Total sentences
---|---|---|---|---|---|---
Asian | 22 | 24 | 14 | 9 | 10 | 79
Black | 30 | 21 | 15 | 7 | 4 | 77
Data not received | 24 | 14 | 10 | 3 | 0 | 51
Mixed Background | 9 | 2 | 0 | 0 | 0 | 11
Other White | 21 | 11 | 4 | 1 | 1 | 38
White British | 132 | 121 | 115 | 179 | 140 | 687

###  3.2 Topics around the sentences of the Black ethnic group

Healthcare Processes and Assessments: The Black ethnic group contributed the
most sentences (29) to this topic, which covers various healthcare processes
and assessments. This suggests a significant focus on the effectiveness and
efficiency of these processes. Concepts such as “Assessment, investigation,
testing, screening”, “Care Planning”, and “Risk assessment” indicate a focus
on the quality and comprehensiveness of patient assessments and care planning.
Keywords like “pathway”, “assessment”, “care”, “plan”, and “review” relate to
the various stages and components of healthcare processes.
Patient Care and Management: The group contributed 24 sentences to this topic,
highlighting their concerns regarding patient care and management. This
reflects an emphasis on ensuring appropriate and effective care for patients.
Concepts such as “Escalation/referral factor”, “Psychological
characteristics”, and “Teamworking” suggest a focus on the various aspects of
patient care, including escalation protocols, mental health, and collaboration
among healthcare professionals. Keywords like “care”, “plan”, “mother”,
“team”, and “treatment” relate to the provision of care and the involvement of
different stakeholders in patient management.
Healthcare Technology and Equipment: The Black ethnic group contributed 21
sentences related to healthcare technology and equipment. This indicates a
moderate level of concern regarding the availability, functionality, and
proper use of technology and equipment in healthcare settings. Concepts such
as “Interpretation”, “Escalation/referral factor”, and “Teamworking” suggest a
focus on the use of technology for tasks like CTG interpretation, as well as
the role of technology in facilitating communication and collaboration among
healthcare professionals. Keywords including “ctg”, “monitoring”, “equipment”,
and “resuscitation” relate to specific technologies and equipment used in
patient care.
COVID-19 Impact on Healthcare: The group contributed 9 sentences related to
the impact of COVID-19 on healthcare. While this topic has fewer sentences
compared to others, it still indicates a notable concern regarding the
challenges and adaptations required in healthcare settings due to the
pandemic. Concepts such as “COVID”, “Communication factor”, and “Teamworking”
suggest a focus on how the pandemic has affected communication, collaboration,
and overall functioning of healthcare teams. Keywords like “resuscitation”,
“staff”, “care”, “ppe”, and “environment” relate to the specific challenges
and changes brought about by COVID-19 in healthcare settings.
Multidisciplinary Care and Communication: The group contributed the least
number of sentences (6) to this topic, which focuses on multidisciplinary care
and communication. Although less prominent than other topics, it still
indicates an awareness of the importance of collaboration and effective
communication among different healthcare professionals. Concepts like “Care
Planning” and “Communication factor” suggest a focus on the coordination of
care and the role of communication in ensuring effective patient management.
Keywords such as “assessment”, “input”, “referrals”, “specialist”, and “meant”
relate to the involvement of various healthcare professionals in patient care
and the communication processes that enable this collaboration.

###  3.3 Topics around the sentences of the Asian ethnic group

Emergency Processes: The Asian ethnic group contributed 22 sentences around
emergency processes, indicating a focus in the responsiveness and management
of emergency situations. This suggests a concern for how emergencies are
handled. Concepts like “Acuity”, “Escalation/referral factor”, and “Situation
awareness” suggest a focus on emergency situations. Keywords such as “ctg”,
“decisions”, “concerns”, “delays”, and “escalation” relate to the management
of urgent cases.
Clinical Operations and Equipment: With 24 sentences, there is a noticeable
focus on clinical operations and the use of medical equipment. This number,
closely following their contributions to emergency processes, reflects an
emphasis on the quality and efficiency of clinical care. Concepts including
“Assessment, investigation, testing, screening”, “Monitoring”, and
“Technologies and Tools-issues” indicate a focus on clinical procedures and
equipment. Keywords like “ctg”, “observation”, “trace”, “auscultation”,
“phone”, and “line” are associated with medical equipment and clinical tasks.
Patient Transfer Logistics: Contributing 14 sentences to this topic, the Asian
ethnic group shows a moderate level of concern regarding the logistics
involved in patient transfer. This interest points to an understanding of the
importance of safe and efficient patient movement within healthcare
facilities. The concepts “Communication factor”, “Documentation”, and
“Teamworking” suggest a focus on the coordination and logistics involved in
patient care and transfer. Keywords such as “samples”, “results”,
“discrepancies”, “handovers”, and “format” relate to the transfer of patient
information and the movement of patients within the healthcare system.
Childbirth Options and Procedures: With only 9 sentences, there is a lower
representation of concerns related to childbirth options and procedures. This
might indicate a lesser focus on or fewer experiences reported regarding
childbirth compared to other topics. The presence of concepts like “Care
Planning” and “Communication factor” in this topic suggests a focus on the
decision-making and communication processes involved in childbirth. Keywords
including “waters”, “leak”, “condition”, and “labour” are specifically related
to childbirth and pregnancy.
Systemic Healthcare Issues: The group contributes 10 sentences on systemic
healthcare issues, suggesting a moderate concern for overarching healthcare
system structures and efficiencies. This indicates an awareness of and concern
for the broader healthcare environment. Concepts such as “Language barrier”
and “Teamworking” in this topic indicate a focus on broader, systemic issues
in healthcare. Keywords like “interpreter”, “trauma”, “intrapartum”,
“postnatal”, and “interpreting” suggest challenges related to communication
and support for patients throughout the healthcare system.

###  3.4 Topics around the sentences of the White British ethnic group

Clinical Assessments and Monitoring: The White British ethnic group
contributed the most sentences (123) to this topic, which covers clinical
assessments, investigations, and monitoring. This suggests a significant focus
on the quality and effectiveness of these processes. Concepts such as
“Assessment, investigation, testing, screening”, “Documentation”, and
“Monitoring” indicate a strong emphasis on the thoroughness and accuracy of
clinical assessments and documentation. Keywords like “clinicians”, “risk”,
“observations”, “examination”, and “history” relate to the various components
and considerations involved in clinical assessments and monitoring.
Communication and Interpersonal Factors: The group contributed 103 sentences
to this topic, highlighting the importance of communication and interpersonal
factors in healthcare. This reflects a focus on ensuring effective
communication among healthcare professionals, patients, and their families.
Concepts such as “Communication factor”, “Teamworking”, and “Psychological
characteristics” suggest a recognition of the role of communication,
collaboration, and emotional support in providing quality care. Keywords like
“birth”, “information”, “family”, “mother”, and “team” relate to the various
stakeholders and contexts in which communication and interpersonal factors are
crucial.
COVID-19 Impact on Healthcare: The White British ethnic group contributed 71
sentences related to the impact of COVID-19 on healthcare. This indicates a
significant concern regarding the challenges and adaptations required in
healthcare settings due to the pandemic. The concept “COVID” is directly
related to this topic, suggesting a focus on the specific effects of the
pandemic on healthcare processes and outcomes. Keywords such as “staff”,
“care”, “maternity”, “equipment”, and “impact” relate to the various aspects
of healthcare that have been affected by COVID-19.
Guidance and Protocol Adherence: The group contributed 65 sentences related to
guidance and protocol adherence. This suggests a notable focus on ensuring
that healthcare professionals follow appropriate guidelines and protocols in
their practice. Concepts such as “Local guidance”, “National guidance”, and
“National and local guidance” directly relate to the use of established
guidelines and protocols in healthcare. Keywords like “guidance”, “line”,
“trust”, “escalation”, and “management” relate to the specific aspects of
healthcare practice that are guided by established protocols and guidelines.
Obstetric Care and Decision-Making: A total of 57 sentences related to
obstetric care and decision-making. This indicates a significant focus on the
specific challenges and considerations involved in providing quality care in
obstetric settings. Concepts such as “Obstetric review”, “Decision error”, and
“Escalation/referral factor” suggest a focus on the critical decision-making
processes and potential pitfalls in obstetric care. Keywords like “birth”,
“pregnancy”, “mother”, “baby”, and “delivery” relate to the specific context
of obstetric care and the key stakeholders involved.

###  3.5 Risks in Topic Modelling with Unbalanced Ethnic Representation

In real-world healthcare data, the under-representation of certain groups
often leads to unbalanced datasets that can significantly impact research
outcomes. Recognising and addressing these risks is crucial for conducting
rigorous and responsible research.
Bias amplification: When a dataset predominantly contains sentences from one
ethnic group, it may overrepresent the perspectives and experiences of that
group, potentially perpetuating stereotypes or inaccuracies. Mitigation:
Enrich the dataset through targeted recruitment from under-represented groups
and employ statistical techniques to adjust for imbalances, ensuring ethical
collection of real, representative data.
Misinterpretation of topics/themes: Data skew may lead to thematic outcomes
that reflect the overrepresented group’s concerns, possibly mischaracterising
crucial topics relevant to other groups. Mitigation: Involve community
representatives and domain experts in the analysis to ensure diverse
perspectives and to validate topics.
Ethical concerns: Analysing unbalanced data without addressing its limitations
may inadvertently marginalise already under-represented voices, compromising
the ethical imperative for equitable representation. Mitigation: Prioritise
ethical oversight and community engagement, acknowledging data limitations
transparently and seeking consent from under-represented groups throughout the
research process.
Loss of valuable insights: A more balanced dataset might reveal unique
experiences and challenges of under-represented groups, offering new
understandings or highlighting areas of concern not visible through the
majority group’s lens. Mitigation: Use existing datasets and conduct
supplementary qualitative research to directly gather insights from under-
represented communities, employing adaptive sampling methods for qualitative
depth.

##  4 Conclusion

This study illustrates the effectiveness of utilising topic modelling and
other NLP techniques to analyse maternity incident investigation reports. It
introduces an analytical framework to highlight disparities in care among
ethnic groups, pointing out the necessity for focused improvement efforts. The
reports underwent data preprocessing, annotation using the SIRch taxonomy, and
topic modelling using LDA to identify unique topics based on different
combinations of concepts and ethnic groups. To ensure the protection of
sensitive data whilst enabling advanced analysis, a combination of offline and
online methods was utilised, with offline processing used for sensitive data
and online processing for non-sensitive data (i.e. keywords of topics as shown
in the Table 2 extract) using the ‘Claude 3 Opus’ language model. Interactive
topic analysis and semantic network visualisation were employed to extract and
display thematic topics and visualise semantic relationships among keywords.
The research acknowledges potential drawbacks of analysing unbalanced
datasets, such as the risk of bias amplification, misinterpretation of
themes/topics, ethical issues, and the possible loss of important insights.
The findings reveal systemic issues within maternity services, setting the
stage for strategies informed by data to prevent incidents and enhance patient
care. Involving stakeholders and experts ensures that the insights from the
analysis are relevant and contribute to targeted measures to improve
healthcare equity and outcomes. A notable limitation of the proposed topic
modelling tool is the absence of Patient and Public Involvement (PPI) in
evaluating its outputs. PPI plays a vital role, ensuring that the derived
solutions are aligned with patient needs and experiences, which could
significantly refine the topic summaries and their applicability in real-world
scenarios. Incorporating input from patients and the public would likely
enhance the model’s transparency and trust, making it more reliable. Future
enhancements will focus on incorporating PPI feedback to improve the
framework’s efficiency in topic modelling, ultimately contributing to better
patient safety and care quality.

####  4.0.1 Acknowledgements

This report is independent research funded by NHSX and The Health Foundation
and it is managed by the National Institute for Health Research (AI_HI200006).
The views expressed in this publication are those of the author(s) and not
necessarily those of the NHSX, The Health Foundation, National Institute for
Health Research, or the Department of Health and Social Care.

####  4.0.2

The authors have no competing interests to declare that are relevant to the
content of this article.

## References

  * [1] Healthcare Safety Investigation Branch: HSIB Maternity Investigation Programme Year in Review 2022/23: Summary of Highlights, Themes and Future Work. https://www.mnsi.org.uk/news/maternity-investigation-programme-year-in-review-202223/ (2023), Accessed: 6th June 2024
  * [2] Ockenden, D.: Findings, conclusions and essential actions from the independent review of maternity services at the Shrewsbury and Telford Hospital NHS Trust. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/1064302/Final-Ockenden-Report-web-accessible.pdf (2022), Accessed: 6th June 2024
  * [3] Royal College of Obstetricians and Gynaecologists: Each Baby Counts: 2020 Final Progress Report. London: RCOG. https://www.rcog.org.uk/media/a4eg2xnm/ebc-2020-final-progress-report.pdf (March 2021), Accessed: 6th June 2024
  * [4] Singh, M.K., Cosma, G., Waterson, P., Back, J., Jun, G.T.: I-SIRch: AI-powered Concept Annotation Tool for Equitable Extraction and Analysis of Safety Insights from Maternity Investigations. Under Review (2023), Preprint available: https://github.com/gcosma/I-SIRchpapers
  * [5] Sutton-Mattocks, Christopher J.: In the Matter of the Inquest Touching the Death of Harry Richford: A Regulation 28 Report - Action to Prevent Future Deaths. https://www.judiciary.uk/wp-content/uploads/2022/10/Harry-Richford-2020-0117-1.pdf (February 2020), Accessed: 6th June 2024


# Risk Scores, Label Bias,
and Everything but the Kitchen Sink

Michael Zanger-Tishler
Harvard University
Julian Nyarko
Stanford University
Sharad Goel
Harvard University

> In designing risk assessment algorithms, many scholars promote a “kitchen
> sink” approach, reasoning that more information yields more accurate
> predictions. We show, however, that this rationale often fails when
> algorithms are trained to predict a proxy of the true outcome, as is
> typically the case. With such “label bias”, one should exclude a feature if
> its correlation with the proxy and its correlation with the true outcome
> have opposite signs, conditional on the other model features. This criterion
> is often satisfied when a feature is weakly correlated with the true
> outcome, and, additionally, that feature and the true outcome are both
> direct causes of the remaining features. For example, due to patterns of
> police deployment, criminal behavior and geography may be weakly correlated
> and direct causes of one’s criminal record, suggesting one should exclude
> geography in criminal risk assessments trained to predict arrest as a proxy
> for behavior.

##  1 Introduction

Risk assessments are central to the allocation of resources and the imposition
of sanctions. In medicine, estimated health risks guide treatment decisions
[1]; in banking, default risk determines whether an applicant should be
granted a loan [2]; in education, the risk of non-completion is an important
factor for college admissions decisions [3]; and in criminal justice,
recidivism risk helps judges decide whether to detain or release a defendant
while their cases proceed [4, 5, 6]. Increasingly, the risk of such adverse
events is estimated with the help of statistical algorithms. In training these
algorithms, there is a widely shared view that the investigator should use as
much data as is available to them [7, 8, 9]. This view rests on the intuition
that more information leads to (weakly) better predictions: If the added data
are informative in estimating risk, then they will improve the performance of
the algorithm, and if the added data do not contain a helpful signal, then
they will be discarded without hurting performance. Proponents of this view
stress that feature importance in the predictive context neither requires nor
implies a causal link between algorithmic inputs and predicted outcomes [8].
Absent the constraints of rigorous causal identification, it is argued that
investigators can remain entirely atheoretical and simply hand all available
data over to the predictive algorithm.

In this paper, we show how “label bias”, present in virtually all real-world
scenarios in which algorithms are deployed today, can invalidate this common
rationale. Label bias occurs when the outcome of interest is not observed
directly, but is instead observed with measurement error. For instance,
although criminal risk assessment tools seek to estimate the risk of future
criminal _behavior_ , we typically only observe whether individuals are
_arrested_ or _convicted_ of a crime. Similarly, tools to estimate health risk
often seek to divert resources to the patients with the most significant
medical _needs_ , but our observations are often limited to medical
_expenditures_. The inclusion of additional features will in general improve
an algorithm’s prediction of the proxy label (e.g., arrest or medical
expenditures), but in the presence of label bias, the additional information
can decrease the quality of predictions for the true label (e.g, criminal
behavior or medical need). Below, we formally demonstrate and empirically
illustrate conditions under which the inclusion of additional features hurts
the predictive performance on the true outcome of interest. Because
researchers rarely have access to the true label, whether or not to include a
particular feature often rests on unverifiable assumptions about the
relationships that gave rise to the proxy label. The findings highlight that
most predictive contexts require investigators to spend significant time and
care in developing a theoretical model of the underlying data generating
process, thus removing one of the most important differentiators between
prediction and causal inference.

Our study contributes to a burgeoning literature examining the use of
algorithmic risk prediction in a variety of domains. These algorithms are
frequently used to predict the risk of adverse events such as future criminal
offending and failure to appear in court [10], the risk of child abuse [11,
12, 13], money laundering [14], students lagging behind in their learning
[15], and the risk of non-payment of loans [2]. They are also used in
situations where organizations or governments are deciding how to allocate
scarce resources such as providing building permits [16], assigning students
to schools [17], assigning high-risk patients to programs providing them more
care [18], and determining who will receive kidney transplants [19]. Further,
corporations are currently using these tools to inform decisions about who
receives information about housing advertisements [20] and employment
opportunities [21]. Algorithmic risk assessment tools can be better than
humans at determining risk [22]. However, scholars continue to critique these
algorithms and study whether and under what conditions they can fairly and
effectively be deployed in society [23, 24, 25, 26].

In addition, our analysis builds on and contributes to a substantial body of
literature examining the impact of label bias in statistical analyses. Prior
work in the social sciences has long focused on the importance of measurement
error for causal studies. Within this literature, a main focus has
traditionally been on examining the importance of measurement error in the
independent variable, which can, at best, attenuate the causal estimates [27,
pp. 320–323], and at worst, bias the coefficients in ways that are difficult
to predict [28]. Less attention has been given to label bias (i.e.,
measurement error in the dependent variable), perhaps because it is often
assumed that proxy labels differ from the true labels by random noise, in
which case one can still obtain unbiased causal estimates [27, pp. 318–320].
Existing research, however, suggests that there is a non-random relationship
between the true and proxy labels across a variety of contexts, such as in the
case of arrest and offending [29]. More recent contributions have considered
the impact of such systematic errors in the labels. For example, Knox et al.
[30] examine the potential for biases to arise in causal estimates when latent
concepts that cannot be directly measured—like political “ideology” and
“democracy”—are approximated by proxy variables constructed from statistical
models. Complementary work in computer science has examined the impact of
label bias in a predictive setting. For instance, although predictive models
may perform well on the proxy label, research has shown they are not
guaranteed to be accurate on the true label if the measurement error between
the true and proxy label is non-random [31]. Similarly, label bias can also
reduce the fairness of these algorithms on the true label [32]. When feasible,
training predictions on the true label rather than a proxy has been shown to
reduce racial inequalities in algorithmic prediction and increase accuracy
[33, 34, 18].

We build on these contributions by explicitly examining how the performance
decrease from label bias interacts with the inclusion of additional predictors
into the model. To establish our results, we begin, in Section 2, by deriving
analytic conditions for when excluding factors in a model trained to predict a
proxy label is guaranteed to improve predictions of the true outcome of
interest. We demonstrate and build intuition for these analytic results using
a stylized example of estimating recidivism risk in the presence of label
bias, where reoffense is the true label of interest and rearrest is the
observed proxy. Then, in Section 3, we turn to two case studies. First, we
consider partially synthetic recidivism data with real rearrest outcomes (the
proxy label) and simulated reoffense outcomes (the true label). This setting
resembles one that many researchers face in practice, where data on the true
label are often prohibitively difficult or impossible to obtain. We show how
different assumptions about how the true label relates to the observed proxy
affect decisions about what predictors to include in the risk assessment
model. Second, we consider a dataset from the health sciences. In targeting
patients for high-risk care management programs, we rely on data by Obermeyer
et al. [18] which contain, among other items, information on both the true
label (healthcare need) and a proxy (healthcare spending). Using this dataset,
we estimate the welfare costs of using a kitchen-sink predictive model instead
of more judiciously selecting a model that accounts for label bias. We
conclude in Section 4 with a discussion of our findings and point out
potential paths forward.

##  2 A Statistical Condition for Excluding Features

To build intuition for how label bias impacts the choice of features in
predictive models, we start with a simplified motivating example from the
criminal justice context. In the United States, after an arrest, a judge will
often decide whether or not to detain the arrested individual based on their
estimated risk to public safety. In practice, this risk is commonly estimated
using statistical risk assessments. The underlying risk models are trained
using information about future arrests and convictions. However, arrests and
convictions are not direct measures of public safety risks. Instead, they
merely act as proxies, making these risk assessment tools susceptible to label
bias.

In Figure 1, we sketch the data-generating process for a stylized, linear
structural equation model (SEM) [35] of arrests and behavior, where we treat
arrests as the observed proxy for unobserved behavior, our true outcome of
interest. The model produces synthetic data on individual-level behavior
($B_{0}$ and $B_{1}$) and arrest ($A_{0}$ and $A_{1}$) outcomes at two time
periods ($t=0$ and $t=1$), as well as the neighborhood ($Z$) in which the
individual resides. Importantly, arrests depend both on behavior and on
neighborhood, reflecting the fact that people who engage in the same behavior
may be arrested at different rates depending on where they live. For example,
Beckett et al. [36] found that the geographic concentration of police
resources in Seattle led to higher arrest rates for Black individuals
delivering drugs compared to white individuals delivering drugs—where the true
racial distribution of those delivering drugs was estimated from survey data
and ethnographic observations. Similarly, Cai et al. [37] found that the
issuance of speeding tickets varied across neighborhoods even after adjusting
for the true, underlying incidence of speeding, as estimated by the movement
of mobile phones.

In this SEM, all of the variables are normally distributed, with mean 0 and
variance 1. We can thus interpret their values as representing the extent to
which individuals differ from the population averages. In the case of
neighborhood ($Z$), we can think of its value as denoting the level of police
enforcement in that area. Further details about the model are provided in the
Appendix.

$Z$Neighborhood$A_{0}$Arrests at $t=0$$A_{1}$Arrests at $t=1$$B_{0}$Behavior
at $t=0$$B_{1}$Behavior at
$t=1$$\alpha$$\alpha$$\beta$$\beta$$\gamma$$\gamma$$\delta$ Figure 1: _The
data-generating process for our stylized example of criminal behavior (true
label) and arrest (proxy label), with observed variables highlighted in
orange._

Using synthetic data generated with this SEM, we train a “complex”, kitchen-
sink model to predict arrests at time $t=1$ ($A_{1}$) based on arrests at time
$t=0$ ($A_{0}$) and neighborhood ($Z$). The more parsimonious, “simple” model
bases its predictions only on arrests at time $t=0$, omitting neighborhood. We
now examine how the performance of the complex and simple models vary for
different values of $\beta$, the parameter that describes the relationship
between neighborhood and behavior, holding the other parameters fixed.111For
this simulation, we set $\alpha=\gamma=\delta=0.4$, though the general pattern
is largely invariant to this choice, as we describe in more detail below.
Across values of $\beta$, the left-hand panel of Figure 2 shows that the
complex model outperforms the simple model—in terms of root mean squared error
(RMSE)—when evaluated on the proxy label. As expected, including more
information reduces error when evaluated on the label used to train the
models, a pattern that has traditionally motivated the inclusion of more
features in predictive models. However, moving to the right-hand panel of
Figure 2, we see that the simple model outperforms the complex model on the
_true_ label for some values of $\beta$. In particular, the simple model
outperforms the complex one for small values of $\beta$, corresponding to a
weak relationship between neighborhood and behavior.

Figure 2: _Performance of simple and complex models trained to predict a proxy
label, when evaluated on the proxy label (left) and the true label (right) for
a range of $\beta$ values. Whereas the complex model outperforms the simple
model on the proxy label, the simple model outperforms the complex model on
the true label for certain values of $\beta$. _

Our SEM illustrates a scenario in which simple models outperform more complex
models due to the presence of label bias. To understand this result, imagine
two individuals, both of whom have the same prior arrest record, but with only
one of them living in a heavily policed neighborhood. Further assume that
where one lives has little impact on criminal behavior (corresponding to small
$\beta$), but that heavier policing increases the chance of being arrested for
an offense. In this case, we can infer that the individual living in the
heavily policed neighborhood engaged in past criminal activity less frequently
than the individual living in the less heavily policed neighborhood. This is
because fewer actual offenses are required to build a given arrest record in
areas of high enforcement. Extrapolating from their past behavior, we would
accordingly expect the individual in the heavily policed area to be less
likely to engage in future criminal behavior. Thus, using information about
one’s neighborhood to predict future arrests (the proxy label) correctly tells
us that the individual living in the heavily policed neighborhood is more
likely to be rearrested, but it incorrectly suggests that individual is also
more likely to engage in future criminal behavior (the true label). So, when
predicting arrests as a proxy for behavior, it is better in this case to
exclude information on one’s neighborhood.

The SEM depicts a specific data-generating process, but the phenomenon we
identify is generalizable. Theorem 1 and Corrolary 1 below establish formal
conditions under which this pattern is guaranteed to occur.

###### Theorem 1

Suppose $Y$ and $Y^{\prime}$ are two arbitrary random variables, where $Y$ is
the “true” outcome of interest and $Y^{\prime}$ is a proxy. For a random
variable $Z$ and a vector of random variables $X=(X_{1},\dots,X_{k})$,
consider the estimators

$\displaystyle\hat{Y}_{X,Z}$ $\displaystyle=\mathbb{E}[Y^{\prime}\mid X,Z],\
\text{and}$ $\displaystyle\hat{Y}_{X}$
$\displaystyle=\mathbb{E}[Y^{\prime}\mid X],$

where $\hat{Y}_{X,Z}$ is the “complex” estimator that uses all available
features, and $\hat{Y}_{X}$ is the “simple” estimator that omits $Z$. If
$\mathrm{Cov}\left(\hat{Y}_{X,Z},Y\mid X\right)\leq 0$, then
$\mathbb{E}\left[\left(\hat{Y}_{X}-Y\right)^{2}\right]\leq\mathbb{E}\left[\left(\hat{Y}_{X,Z}-Y\right)^{2}\right]$,
meaning the simple estimator weakly outperforms the complex estimator.
Further, if $\mathbb{E}\left[\mathrm{Var}\left(\hat{Y}_{X,Z}\mid
X\right)\right]\neq 0$, then the simple estimator strictly outperforms the
complex estimator.

In the setting of Theorem 1, one seeks to estimate a true outcome of interest
$Y$, and must choose between two different estimators designed to predict the
proxy label $Y^{\prime}$. The first, “complex” estimator ($\hat{Y}_{X,Z}$)
uses both $X$ and $Z$ to predict $Y^{\prime}$, whereas the second
($\hat{Y}_{X}$) uses only $X$. The theorem shows that if, conditional on $X$,
the true label ($Y$) is negatively correlated with the complex estimator
($\hat{Y}_{X,Z}$), then the simple model generally outperforms the complex
estimator on the true outcome of interest. If, alternatively, the true and
proxy labels differ only by additive, independent noise, then Proposition 1 in
the Appendix shows that including more information when predicting the proxy
label will in general improve predictive performance on the true label. Thus,
in the absence of systematic measurement error—including the case where there
is no measurement error—that result confirms the conventional wisdom that more
information is better .

When the complex estimator $\hat{Y}_{X,Z}$ is linear in $Z$, Corollary 1
establishes a simpler condition under which performance increases by omitting
information. Specifically, if, conditional on $X$, $Z$ is positively
correlated with true label $Y$ but negatively correlated with the proxy label
$Y^{\prime}$ (or vice versa), then omitting $Z$ when predicting the proxy
label will in general improve performance on the true outcome of interest.

###### Corollary 1

In the setting of Theorem 1, suppose $\hat{Y}_{X,Z}$ is linear in $Z$, i.e.,
$\hat{Y}_{X,Z}=f(X)+g(X)\cdot Z$ for some functions $f$ and $g$. If

$\mathrm{sign}\left(\mathrm{Cov}\left(Y,Z\mid
X\right)\right)=-\mathrm{sign}\left(\mathrm{Cov}\left(Y^{\prime},Z\mid
X\right)\right),$

then
$\mathbb{E}\left[\left(\hat{Y}_{X}-Y\right)^{2}\right]\leq\mathbb{E}\left[\left(\hat{Y}_{X,Z}-Y\right)^{2}\right]$.
Further, if $\mathbb{E}\left[\mathrm{Var}\left(\hat{Y}_{X,Z}\mid
X\right)\right]\neq 0$, then the simple estimator strictly outperforms the
complex estimator.

The linearity assumption of Corollary 1 holds in a variety of settings. In
particular, as described in the Appendix, it holds when $Y^{\prime}$, $X$, and
$Z$ are jointly multivariate normal, as is the case in our SEM above. To apply
the corollary, one needs information on the correlations of $Y$ and $Z$ and of
$Y^{\prime}$ and $Z$, conditional on $X$. The former involves directly
observed quantities—the proxy label and the potential predictors—and so, in
practice, can be computed from data. For our stylized SEM, we show in the
Appendix that this correlation is positive for all (non-degenerate) parameter
choices, meaning that neighborhood ($Z$) is positively correlated with future
arrests ($A_{1}$), conditional on past arrests ($A_{0}$). The second
conditional correlation we must consider when applying Corollary 1—the
correlation between $Y$ and $Z$, conditional on $X$—is not typically directly
observed, as it depends on the true label $Y$. Understanding its sign thus
involves assumptions about how the true label is related to the predictors $Z$
and $X$. For our SEM, we show in the Appendix that this correlation is
negative for small values of $\beta$. That is, when $\beta$ is small,
neighborhood ($Z$) and future behavior ($B_{1}$) are negatively correlated
conditional on past arrests ($A_{0}$). Intuitively, this is because $A_{0}$ is
a collider, and so when we fix its value, increasing $Z$ requires decreasing
$B_{0}$, which in turn decreases $B_{1}$. Thus, for small values of $\beta$,
omitting neighborhood when predicting the proxy label improves performance on
the true label, as shown in Figure 3.

##  3 Case Studies

To better understand the practical implications of our results, we now turn to
two real-world datasets. The first allows us to further consider criminal risk
assessments, adding additional realism to our stylized SEM above; the second
dataset comes from the medical domain, where the goal of the risk assessment
we consider is to identify patients with complex healthcare needs.

###  3.1 Criminal risk assessments

Continuing with our running example studying arrest and criminal behavior, we
use data on individuals from a major U.S. county who were arrested for a
felony offense between 2013 and 2019. For simplicity, we limit the sample to
the 25,918 cases where the individuals’ race was identified as either Black or
non-Hispanic white. The dataset includes further details on each case,
including information on the charges, the location, date and time of the
incident, and the criminal history of the arrested individual. In addition,
the dataset contains information on future rearrests, which we use as our
proxy label for future offenses. Using these data, we fit simple and complex
models trained on the proxy label (future arrests). We then examine model
performance on the true label (future criminal offenses, which we simulate, as
described below, since they are not directly observed). Our “complex” model
includes three features: the age of the arrested individual; the number of
times the individual was previously arrested; and whether or not the arrest
occurred in a “high policing” area (i.e., a police district accounting for
disproportionately high numbers of arrests). Our “simple” model includes age
and number of past arrests, but not location information—similar to many
commonly used criminal risk assessment tools.

This example mirrors many instances of label bias in the real world, as it is
difficult, and perhaps impossible, to directly estimate the risk of “true”
offending [38]. This is in part because criminal behavior that is not reported
to the police will not be included in administrative records. We thus simulate
offending outcomes under a range of data-generating processes, and then
examine how assumptions about criminal behavior affect model performance after
including or omitting location information. In particular, for a fixed value
of $\rho\in\mathbb{R}$, describing the impact of neighborhood on criminal
behavior, we assume that each individual in our dataset commits a future
offense with the following probability:

$\Pr(B_{1}=1)=\text{logit}^{-1}\left(-1-\frac{1}{100}X_{\text{age}}+\frac{1}{2}A_{0}+\rho
Z\right),$

where $B_{1}$ indicates future criminal behavior (our true label),
$X_{\text{age}}$ is the arrested individual’s age, $A_{0}$ is the number of
times they were previously arrested, and $Z$ indicates whether the arrest took
place in a high-policing area. The intercept and the coefficients for $A_{0}$
and $X_{\text{age}}$ were selected to approximate the coefficients from a
regression of future arrests on age and past arrests in our data.

Figure 3: _Performance of simple (age and past arrests) and complex (age, past
arrests, and neighborhood) models trained to predict future arrests (the proxy
label), evaluated on future criminal behavior (the true label). Because the
future criminal behavior is not directly observable, the plot shows results
for synthetic outcomes generated under a range of data-generating processes
parameterized by $\rho$, the hypothesized relationship between neighborhood
and future criminal behavior._

Based on the data-generating process described above, we now evaluate the
ability of our simple and complex risk assessment models to predict the
synthetic true label, future criminal behavior. We evaluate model performance
in terms of AUC, as the outcome is binary.222AUC is a common measure of
performance in the machine learning community when considering binary
outcomes. Given a random individual who engaged in future criminal activity
and a random individual who did not, the AUC of a risk assessment model is the
probability that the model correctly identifies the individual in the pair who
engaged in criminal activity. Our formal theoretical results are stated in
terms of RMSE, but this example and our subsequent example show that the
general pattern and intuition extend to other popular evaluation metrics.
Figure 3 shows that the simple model outperforms the complex model on the true
label when $\rho$ is negative, and the complex model outperforms the simple
model when $\rho$ is sufficiently positive. Given two arrested individuals who
are the same age and have the same number of past arrests, negative values of
$\rho$ indicate that the individual who was arrested in the high-policing area
is the less likely of the pair to engage in future criminal behavior. That
pattern is akin to what we saw in our stylized SEM depicted in Figure 1.
Accordingly, to the extent that one believes the hypothesized data-generating
process with negative $\rho$ is a sufficiently accurate description of
criminal behavior, it is better to exclude neighborhood information when
training risk assessment tools on the proxy label, future arrests.

###  3.2 Identifying high-needs patients

We continue by applying our results to a well-known case of label bias in the
literature, that of a commercial risk assessment tool that health systems rely
on to target patients for “high-risk care management” programs [18]. These
programs seek to enroll patients with complex medical needs, and subsequently
provide them with a higher level of care. Because these programs are capacity
constrained, the role of statistical risk assessments in this case is to
accurately identify patients who would benefit the most from the additional
care. In practice, though, the risk assessment algorithms are often designed
to predict future medical expenditures, a proxy for medical need as the true
outcome of interest. Analyzing these algorithms, Obermeyer et al. [18]
conclude that, due to label bias, Black patients are less likely to be
enrolled in the program than white patients with the same level of medical
need. This is because unequal access to healthcare means that white
individuals are more likely to seek medical treatment—and accordingly incur
higher medical costs—than equally sick racial minorities.

Obermeyer et al. [18] highlight the importance of appropriately selecting the
target of prediction, and illustrate the accuracy and equity gains one can
achieve by switching from predicting expenditures to a more direct measure of
medical need. Here we revisit the problem, and investigate how the choice of
risk factors used to identify patients impacts enrollment decisions. To do so,
we start with the data released by Obermeyer et al. [18], which include
detailed information on patient demographics (sex, race, and age), current and
future health, and past and future medical expenditures.333Obermeyer et al.
[18] released a synthetic dataset, with variables having the same conditional
distributions as those in the original dataset, using the synthpop package in
R (the original data cannot be released in order to protect patient privacy).
The data are available at: https://gitlab.com/labsysmed/dissecting-bias). We
then train simple and complex models on the proxy label, future medical costs.
Our complex model includes all information available at the time of the
enrollment decision (i.e., patient demographics, current health, and past
medical expenditures); our simple model includes only current health,
excluding past medical medical costs and demographic variables. In the end,
the complex model includes 150 predictors, and the simple model includes 128
predictors.

Figure 4: _Enrollment of high needs patients (left) and demographic
composition of enrolled patients (right) under the simple and complex models
for a range of program capacities._

Finally, we evaluate both models on their ability to predict whether a
patient, in the subsequent year, is found to suffer from at least three
chronic diseases—a measure of future health need identified by Obermeyer et
al. [18]. The left-hand panel of Figure 4 shows the number of high-needs
patients enrolled under the simple and complex models at different enrollment
capacities, where the patients with highest estimated risk under the
respective models are enrolled in the program. At each capacity level, the
simple model outperforms the complex model in identifying more high-needs
patients. Additionally, as shown in the right-hand panel of Figure 4, the
simple model enrolls more Black patients than the complex model at every
capacity level. This pattern stems from the simple model prioritizing patients
with high expected medical needs over patients with high expected medical
expenditures—the latter population being disproportionately white. Thus, if
one only has access to a proxy label, systematically excluding predictors in a
risk assessment tool can improve both the accuracy and equity of the
instrument.

##  4 Conclusion

In building predictive models, the traditional guidance is to include all
available information to maximize performance. But, as we have shown, a more
judicious selection of features can lead to better model performance in the
presence of label bias. Because the true label of interest is often not
readily available, it raises the question of what examiners should and can do
to mitigate the negative consequences from taking a kitchen-sink approach to
prediction. The examples we have discussed highlight several approaches that
vary in their appropriateness based on data availability and understanding of
the underlying data-generating process.

Most directly, Obermeyer et al. [18] illustrate how some instances of label
bias can be addressed simply by making a more concentrated effort to collect
data on the true label of interest. If such an effort is generally possible,
but prohibitively costly, investigators should consider whether the true label
of interest can be obtained for a smaller subset of the population. This
subset, even if it is not sufficiently large to train models predicting the
true label, might still be used to explore how the selection of features
affects model performance on the true label. If obtaining the true label is
impossible, but investigators have access to a wealth of other features, one
may simulate the true label of interest. In doing so, researchers should use
their domain-specific knowledge to make reasonable assumptions about the
relationship between the true label of interest and the features in question.
We illustrated this process using felony offense data. Importantly,
investigators need not constrain themselves to one particular relationship
between the true label and the features, but can instead assess the
sensitivity of feature selection to label bias across a wide range of
plausible assumptions. Finally, investigators can make additional theoretical
assumptions about the data-generating process in order to determine how label
bias affects the choice of risk factors in a specific application—as we did in
our healthcare example. As shown in that example, caution is particularly
warranted for features that do not appear to be directly risk relevant. These
features often yield little improvement on the true outcome of interest, and
raise the likelihood that performance may decrease or that their inclusion may
exacerbate disparities.

More generally, our findings suggest, in contrast to conventional wisdom, that
one cannot entirely divorce the predictive enterprise from theoretical
considerations. Instead, a successful deployment of predictive tools often
rests on the plausibility of the assumptions about the underlying processes
that give rise to the observed data, highlighting the continued utility of
domain-specific expertise in the predictive context.

## References

  * [1] Sendhil Mullainathan and Ziad Obermeyer.  Diagnosing physician error: A machine learning approach to low-value health care.  The Quarterly Journal of Economics, 137(2):679–727, 2022.
  * [2] Martin Leo, Suneel Sharma, and Koilakuntla Maddulety.  Machine learning in banking risk management: A literature review.  Risks, 7(1):29, 2019.
  * [3] Lovenoor Aulck, Dev Nambi, Nishant Velagapudi, Joshua Blumenstock, and Jevin West.  Mining university registrar records to predict first-year undergraduate attrition.  International Educational Data Mining Society, 2019.
  * [4] Alexandra Chouldechova.  Fair prediction with disparate impact: A study of bias in recidivism prediction instruments.  Big Data, 5(2):153–163, 2017.
  * [5] Julia Dressel and Hany Farid.  The accuracy, fairness, and limits of predicting recidivism.  Science Advances, 4(1):eaao5580, 2018.
  * [6] Zhiyuan Lin, Jongbin Jung, Sharad Goel, and Jennifer Skeem.  The limits of human predictions of recidivism.  Science Advances, 6(7):eaaz0652, 2020.
  * [7] Richard Berk.  Machine Learning Risk Assessments in Criminal Justice Settings.  Springer, 2019.
  * [8] Charles F Manski.  Patient-centered appraisal of race-free clinical risk assessment.  Health Economics, 31(10):2109–2114, 2022.
  * [9] Charles F Manski, John Mullahy, and Atheendar Venkataramani.  Using measures of race to make clinical predictions: Decision making, patient health, and fairness.  Technical report, National Bureau of Economic Research, 2022.
  * [10] Kosuke Imai, Zhichao Jiang, D James Greiner, Ryan Halen, and Sooahn Shin.  Experimental evaluation of algorithm-assisted human decision-making: Application to pretrial public safety assessment.  2022\.
  * [11] Anna Brown, Alexandra Chouldechova, Emily Putnam-Hornstein, Andrew Tobin, and Rhema Vaithianathan.  Toward algorithmic accountability in public services: A qualitative study of affected community perspectives on algorithmic decision-making in child welfare services.  In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pages 1–12, 2019.
  * [12] Alexandra Chouldechova, Diana Benavides-Prado, Oleksandr Fialko, and Rhema Vaithianathan.  A case study of algorithm-assisted decision making in child maltreatment hotline screening decisions.  In Conference on Fairness, Accountability and Transparency, pages 134–148, 2018.
  * [13] Ravi Shroff.  Predictive analytics for city agencies: Lessons from children’s services.  Big Data, 5(3):189–196, 2017.
  * [14] Yan Zhang and Peter Trubey.  Machine learning and sampling scheme: An empirical study of money laundering detection.  Computational Economics, 54:1043–1063, 2019.
  * [15] Lindsay Cattell and Julie Bruch.  Identifying students at risk using prior performance versus a machine learning algorithm.  Technical report, U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Mid-Atlantic, 2021.
  * [16] Viktor Mayer-Schönberger and Kenneth Cukier.  Big Data: A Revolution That Will Transform How We Live, Work, and Think.  Houghton Mifflin Harcourt, 2013.
  * [17] Maxwell Allman, Itai Ashlagi, Irene Lo, Juliette Love, Katherine Mentzer, Lulabel Ruiz-Setz, and Henry O’Connell.  Designing school choice for diversity in the San Francisco Unified School District.  In Proceedings of the 23rd ACM Conference on Economics and Computation, pages 290–291, 2022.
  * [18] Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan.  Dissecting racial bias in an algorithm used to manage the health of populations.  Science, 366(6464):447–453, 2019.
  * [19] John J Friedewald, Ciara J Samana, Bertram L Kasiske, Ajay K Israni, Darren Stewart, Wida Cherikh, and Richard N Formica.  The kidney allocation system.  Surgical Clinics, 93(6):1395–1406, 2013.
  * [20] Till Speicher, Muhammad Ali, Giridhari Venkatadri, Filipe Nunes Ribeiro, George Arvanitakis, Fabrício Benevenuto, Krishna P. Gummadi, Patrick Loiseau, and Alan Mislove.  Potential for discrimination in online targeted advertising.  In Proceedings of the 1st Conference on Fairness, Accountability and Transparency, volume 81 of Proceedings of Machine Learning Research, pages 5–19. PMLR, 23–24 Feb 2018.
  * [21] Anja Lambrecht and Catherine Tucker.  Algorithmic bias? An empirical study of apparent gender-based discrimination in the display of stem career ads.  Management Science, 65(7):2966–2981, 2019.
  * [22] Sharad Goel, Ravi Shroff, Jennifer Skeem, and Christopher Slobogin.  The accuracy, equity, and jurisprudence of criminal risk assessment.  In Research Handbook on Big Data Law, pages 9–28. Edward Elgar Publishing, 2021.
  * [23] Angelina Wang, Sayash Kapoor, Solon Barocas, and Arvind Narayanan.  Against predictive optimization: On the legitimacy of decision-making algorithms that optimize predictive accuracy.  Available at SSRN, 2022.
  * [24] Sam Corbett-Davies, Johann Gaebler, Hamed Nilforoshan, Ravi Shroff, and Sharad Goel.  The measure and mismeasure of fairness.  arXiv preprint arXiv:1808.00023, 2023.
  * [25] Alexandra Chouldechova and Aaron Roth.  A snapshot of the frontiers of fairness in machine learning.  Communications of the ACM, 63(5):82–89, 2020.
  * [26] Alex Chohlas-Wood, Madison Coots, Sharad Goel, and Julian Nyarko.  Designing equitable algorithms.  arXiv preprint arXiv:2302.09157, 2023.
  * [27] Jeffrey M Wooldridge.  Introductory Econometrics: A Modern Approach.  Cengage Learning, 2015.
  * [28] Aaron Chalfin and Justin McCrary.  Are U.S. cities underpoliced? Theory and evidence.  Review of Economics and Statistics, 100(1):167–186, 2018.
  * [29] Riccardo Fogliato, Alice Xiang, Zachary Lipton, Daniel Nagin, and Alexandra Chouldechova.  On the validity of arrest as a proxy for offense: Race and the likelihood of arrest for violent crimes.  In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pages 100–111, 2021.
  * [30] Dean Knox, Christopher Lucas, and Wendy K Tam Cho.  Testing causal theories with learned proxies.  Annual Review of Political Science, 25:419–441, 2022.
  * [31] Jialu Wang, Yang Liu, and Caleb Levy.  Fair classification with group-dependent label noise.  In Proceedings of the 2021 ACM conference on Fairness, Accountability, and Transparency, pages 526–536, 2021.
  * [32] Riccardo Fogliato, Alexandra Chouldechova, and Max G’Sell.  Fairness evaluation in presence of biased noisy labels.  In International Conference on Artificial Intelligence and Statistics, pages 2325–2336. PMLR, 2020.
  * [33] Emma Pierson, David M Cutler, Jure Leskovec, Sendhil Mullainathan, and Ziad Obermeyer.  An algorithmic approach to reducing unexplained pain disparities in underserved populations.  Nature Medicine, 27(1):136–140, 2021.
  * [34] Sendhil Mullainathan and Ziad Obermeyer.  On the inequity of predicting a while hoping for b.  In AEA Papers and Proceedings, volume 111, pages 37–42, 2021.
  * [35] Judea Pearl.  Linear models: A useful “microscope” for causal analysis.  Journal of Causal Inference, 1(1):155–170, 2013.
  * [36] Katherine Beckett, Kris Nyrop, and Lori Pfingst.  Race, drugs, and policing: Understanding disparities in drug delivery arrests.  Criminology, 44(1):105–137, 2006.
  * [37] William Cai, Johann Gaebler, Justin Kaashoek, Lisa Pinals, Samuel Madden, and Sharad Goel.  Measuring racial and ethnic disparities in traffic enforcement with large-scale telematics data.  PNAS Nexus, 1(4):pgac144, 2022.
  * [38] Albert D Biderman and Albert J Reiss Jr.  On exploring the “dark figure” of crime.  The Annals of the American Academy of Political and Social Science, 374(1):1–15, 1967.
  * [39] Sewall Wright.  Systems of mating. I. The biometric relations between parent and offspring.  Genetics, 6(2):111, 1921.
  * [40] Morris L Eaton.  Multivariate Statistics: A Vector Space Approach. John Wiley & Sons, Inc., 1983.

## Acknowledgements

We thank Avi Feller, Johann Gaebler, Talia Gillis, and Josh Grossman for
helpful feedback and conversations.

##  Appendix A Proof of Theorem 1

For any random variable $\hat{Y}$,

$\mathbb{E}\left[\left(\hat{Y}-Y\right)^{2}\right]=\mathbb{E}\left[Y^{2}\right]+\mathbb{E}\left[\hat{Y}^{2}\right]-2\mathbb{E}\left[Y\cdot\hat{Y}\right],$

and so,

$\displaystyle\begin{split}&\mathbb{E}\left[\left(\hat{Y}_{X,Z}-Y\right)^{2}\right]-\mathbb{E}\left[\left(\hat{Y}_{X}-Y\right)^{2}\right]\\\
&\hskip
72.26999pt=\left(\mathbb{E}\left[\hat{Y}_{X,Z}^{2}\right]-\mathbb{E}\left[\hat{Y}_{X}^{2}\right]\right)+2\Big{(}\mathbb{E}\left[Y\cdot\hat{Y}_{X}\right]-\mathbb{E}\left[Y\cdot\hat{Y}_{X,K}\right]\Big{)}.\end{split}$
(1)

We will show, in turn, that each of the summands on right-hand side of Eq. (1)
are non-negative—and that the former is strictly positive when
$\mathbb{E}\left[\mathrm{Var}\left(\hat{Y}_{X,Z}\mid X\right)\right]\neq 0$.
To start, we note that

$\displaystyle\begin{split}\mathbb{E}\left[\hat{Y}_{X,Z}^{2}\mid
X\right]&\geq\left(\mathbb{E}\left[\hat{Y}_{X,Z}\mid X\right]\right)^{2}\\\
&=\left(\mathbb{E}\left[\mathbb{E}[Y^{\prime}\mid X,Z]\mid
X\right]\right)^{2}\\\ &=\left(\mathbb{E}\left[Y^{\prime}\mid
X\right]\right)^{2}\\\ &=\hat{Y}_{X}^{2},\end{split}$ (2)

where the first line follows from Jensen’s inequality, and the second equality
follows from the law of iterated expectations. As a result, by another
application of the law of iterated expectations,

$\displaystyle\begin{split}\mathbb{E}\left[\hat{Y}_{X,Z}^{2}\right]&=\mathbb{E}\left[\mathbb{E}\left[\hat{Y}_{X,Z}^{2}\mid
X\right]\right]\\\ &\geq\mathbb{E}\left[\hat{Y}_{X}^{2}\right],\end{split}$
(3)

showing that the first summand on right-hand side of Eq. (1) is non-negative.
Further, the inequality in Eq. (2) is strict on the set where
$\mathrm{Var}\left(\hat{Y}_{X,Z}\mid X\right)\neq 0$. Consequently, if
$\mathbb{E}\left[\mathrm{Var}\left(\hat{Y}_{X,Z}\mid X\right)\right]\neq 0$,
then $\mathrm{Var}\left(\hat{Y}_{X,Z}\mid X\right)\neq 0$ on a set of positive
measure, and so the inequality Eq. (3) is likewise strict.

Now, turning to the second summand, we have

$\displaystyle\mathbb{E}\left[Y\cdot\hat{Y}_{X,Z}\right]$
$\displaystyle=\mathbb{E}\left[\mathbb{E}\left[Y\cdot\hat{Y}_{X,Z}\mid
X\right]\right]$
$\displaystyle\leq\mathbb{E}\left[\mathbb{E}\left[\hat{Y}_{X,Z}\mid
X\right]\cdot\mathbb{E}\left[Y\mid X\right]\right]$
$\displaystyle=\mathbb{E}\left[\hat{Y}_{X}\cdot\mathbb{E}\left[Y\mid
X\right]\right]$
$\displaystyle=\mathbb{E}\left[\mathbb{E}\left[Y\cdot\hat{Y}_{X}\mid
X\right]\right]$ $\displaystyle=\mathbb{E}\left[Y\cdot\hat{Y}_{X}\right],$

where we repeatedly applied the law of iterated expectations, used the fact
that $\hat{Y}_{X}$ is measurable with respect to $X$ to establish the
penultimate equality, and the assumption of the theorem that
$\mathrm{Cov}(\hat{Y}_{X,Z},Y\mid X)\leq 0$ to establish the inequality. The
above shows that the second summand on the right-hand side of Eq. (1) is non-
negative. Since we previously showed that the first summand is non-negative
(and strictly positive under the additional assumption), this proves the
result.

$\hbox{}\nobreak\hfill\square$

##  Appendix B Proof of Corollary 1

By Theorem 1, it is sufficient to show that $\mathrm{Cov}(\hat{Y}_{X,Z},Y\mid
X)\leq 0$. We start by noting that

$\displaystyle\mathrm{Cov}(\hat{Y}_{X,Z},Y\mid X)$
$\displaystyle=\mathrm{Cov}(f(X)+g(X)\cdot Z,Y\mid X)$
$\displaystyle=g(X)\cdot\mathrm{Cov}(Y,Z\mid X),$

and so, by the assumption of the theorem,

$\mathrm{sign}\left(\mathrm{Cov}\left(\hat{Y}_{X,Z},Y\mid
X\right)\right)=-\mathrm{sign}\left(g(X)\cdot\mathrm{Cov}\left(Y^{\prime},Z\mid
X\right)\right).$ (4)

Now, by repeatedly applying the law of iterated expectations, we have

$\displaystyle\mathbb{E}\left[Z\cdot Y^{\prime}\mid X\right]$
$\displaystyle=\mathbb{E}\left[\mathbb{E}\left[Z\cdot Y^{\prime}\mid
X,Z\right]\mid X\right]$
$\displaystyle=\mathbb{E}\left[Z\cdot\mathbb{E}\left[Y^{\prime}\mid
X,Z\right]\mid X\right]$
$\displaystyle=\mathbb{E}\left[Z\cdot\hat{Y}_{X,Z}\mid X\right]$
$\displaystyle=f(X)\cdot\mathbb{E}[Z\mid X]+g(X)\cdot\mathbb{E}\left[Z^{2}\mid
X\right].$

Similarly, we have

$\displaystyle\mathbb{E}[Y^{\prime}\mid X]$
$\displaystyle=\mathbb{E}[\mathbb{E}[Y^{\prime}\mid X,Z]\mid X]$
$\displaystyle=\mathbb{E}[\hat{Y}_{X,Z}\mid X]$
$\displaystyle=f(X)+g(X)\cdot\mathbb{E}[Z\mid X].$

Putting the above together, we get

$\displaystyle\mathrm{Cov}\left(Y^{\prime},Z\mid X\right)$
$\displaystyle=\mathbb{E}\left[Z\cdot Y^{\prime}\mid
X\right]-\mathbb{E}[Y^{\prime}\mid X]\cdot\mathbb{E}[Z\mid X]$
$\displaystyle=g(X)\left(\mathbb{E}\left[Z^{2}\mid
X\right]-\mathbb{E}\left[Z\mid X\right]^{2}\right)$
$\displaystyle=g(X)\cdot\mathrm{Var}(Z\mid X).$

Finally, by Eq. (4),

$\displaystyle\mathrm{sign}\left(\mathrm{Cov}\left(\hat{Y}_{X,Z},Y\mid
X\right)\right)$
$\displaystyle=-\mathrm{sign}\left(g(X)^{2}\cdot\mathrm{Var}(Z\mid X)\right)$
$\displaystyle\leq 0,$

establishing the result.

$\hbox{}\nobreak\hfill\square$

##  Appendix C Kitchen-Sink Models and Independent Noise

When the proxy label $Y^{\prime}$ and the true label $Y$ simply differ by
additive, independent noise, then it is advantageous to use all available
information when constructing risk scores. The following proposition
formalizes this statement.

###### Proposition 1

In the setting of Theorem 1, suppose $Y^{\prime}=Y+S$ where
$S\perp\\!\\!\\!\perp X,Z$. Then

$\mathbb{E}\left[\left(\hat{Y}_{X,Z}-Y\right)^{2}\right]\leq\mathbb{E}\left[\left(\hat{Y}_{X}-Y\right)^{2}\right].$

_Proof._ First note that

$\displaystyle\hat{Y}_{X,Z}$ $\displaystyle=\mathbb{E}[Y\mid
X,Z]+\mathbb{E}[S\mid X,Z]$ $\displaystyle=\mathbb{E}[Y\mid
X,Z]+\mathbb{E}[S],$

where the second equality uses the independence assumption. Similarly,

$\displaystyle\hat{Y}_{X}$ $\displaystyle=\mathbb{E}[Y\mid X]+\mathbb{E}[S\mid
X]$ $\displaystyle=\mathbb{E}[Y\mid X]+\mathbb{E}[S].$

Now, using the notation $Y_{X,Z}=\mathbb{E}[Y\mid X,Z]$ and
$Y_{X}=\mathbb{E}[Y\mid X]$, we have

$\displaystyle\mathbb{E}\left[\left(\hat{Y}_{X,Z}-Y\right)^{2}\right]-\mathbb{E}\left[\left(\hat{Y}_{X}-Y\right)^{2}\right]$
$\displaystyle=\mathbb{E}\left[\big{(}Y_{X,Z}-Y+\mathbb{E}[S]\big{)}^{2}\right]-\mathbb{E}\left[\big{(}Y_{X}-Y+\mathbb{E}[S]\big{)}^{2}\right]$
$\displaystyle=\mathbb{E}\left[\big{(}Y_{X,Z}-Y\big{)}^{2}\right]-\mathbb{E}\left[\big{(}Y_{X}-Y\big{)}^{2}\right]+2\mathbb{E}[S]\big{(}\mathbb{E}[Y_{X,Z}-Y]-\mathbb{E}[Y_{X}-Y]\big{)}$
$\displaystyle=\mathbb{E}\left[\big{(}Y_{X,Z}-Y\big{)}^{2}\right]-\mathbb{E}\left[\big{(}Y_{X}-Y\big{)}^{2}\right]$
$\displaystyle=\mathbb{E}\left[\mathbb{E}\left[\big{(}Y_{X,Z}-Y\big{)}^{2}\mid
X,Z\right]\right]-\mathbb{E}\left[\mathbb{E}\left[\big{(}Y_{X}-Y\big{)}^{2}\mid
X,Z\right]\right],$

where the third equality follows from the fact that
$\mathbb{E}[Y_{X,Z}]=\mathbb{E}[Y_{X}]=\mathbb{E}[Y]$, and the last equality
follows from the law of iterated expectations. Finally, since

$\arg\min_{c}\mathbb{E}\left[\big{(}c-Y\big{)}^{2}\mid X,Z\right]=Y_{X,Z},$

we have that

$\mathbb{E}\left[\big{(}Y_{X,Z}-Y\big{)}^{2}\mid
X,Z\right]-\mathbb{E}\left[\big{(}Y_{X}-Y\big{)}^{2}\mid X,Z\right]\leq 0,$

establishing the result.

$\hbox{}\nobreak\hfill\square$

##  Appendix D A Stylized Model of Arrest and Behavior

We formally describe and analyze the SEM depicted in Figure 1. Our model has
three independent exogenous variables $U_{Z}=\mathrm{N}(0,\sigma_{Z}^{2})$,
$U_{A_{0}}=\mathrm{N}(0,\sigma_{A}^{2})$, and
$U_{A_{1}}=\mathrm{N}(0,\sigma_{A}^{2})$. We additionally have two correlated
exogenous variables $U_{B_{0}}=\mathrm{N}(0,\sigma_{B}^{2})$ and
$U_{B_{1}}=\mathrm{N}(0,\sigma_{B}^{2})$ that are independent of the first
three, with $\mathrm{Cov}(U_{B_{0}},U_{B_{1}})=\delta\geq 0$. Now, for non-
negative constants $\alpha$, $\beta$, and $\gamma$, the key variables in the
model are generated by the following linear structural equations:

$\displaystyle\begin{split}Z&=U_{Z},\\\ B_{0}&=\beta Z+U_{B_{0}},\\\
B_{1}&=\beta Z+U_{B_{1}},\\\ A_{0}&=\alpha Z+\gamma B_{0}+U_{A_{0}},\\\
A_{1}&=\alpha Z+\gamma B_{1}+U_{A_{1}}.\\\ \end{split}$ (5)

We set the variances of the exogenous variables ($\sigma_{Z}^{2}$,
$\sigma_{A}^{2}$, and $\sigma_{B}^{2}$) in a manner that ensures that the
remaining variables ($Z$, $B_{0}$, $B_{1}$, $A_{0}$, and $A_{1}$) are
standardized, meaning they have mean 0 and variance 1—we show how to do this
below. We can thus interpret their values as representing the extent to which
individuals differ from the population averages. In the case of neighborhood
($Z$), we can think of its value as denoting the level of police enforcement
in an area.

To start, we set $\sigma_{Z}^{2}=1$, which ensures $\mathrm{Var}(Z)=1$. Now,
since $Z\perp\\!\\!\\!\perp U_{B_{0}}$, we have that
$\mathrm{Var}(B_{0})=\beta^{2}+\sigma_{B}^{2}$. Consequently, setting
$\sigma_{B}^{2}=1-\beta^{2}$ ensures that $\mathrm{Var}(B_{0})=1$ (and,
similarly, that $\mathrm{Var}(B_{1})=1$). Finally, as above,
$\mathrm{Var}(A_{0})=\alpha^{2}+\gamma^{2}+\sigma_{A}^{2}+2\alpha\gamma\mathrm{Cov}(Z,B_{0})$.
One especially nice aspect of linear graphical models is that the covariance
between any two variables can be immediately computed from the edge weights
via the the Wright rules [39, 35]. Specifically, when the nodes are
standardized to have variance 1, then the covariance between any two variables
in the graph is the sum, over all $d$-connected paths between the variables,
of the product of the edge weights along the path. A path is $d$-connected if
it does not pass through any colliders (i.e., nodes with head-to-head arrows
along the path). To compute $\mathrm{Cov}(Z,B_{0})$, observe that the only
$d$-connected path between $Z$ and $B_{0}$ is the direct path from $Z$ to
$B_{0}$, having edge weight $\beta$. As a result,
$\mathrm{Cov}(Z,B_{0})=\beta$, meaning that setting
$\sigma_{A}^{2}=1-\alpha^{2}-\gamma^{2}-2\alpha\beta\gamma$ ensures that
$A_{0}$ (and, analogously, $A_{1}$) have unit variance. Recapping, we have

$\displaystyle\begin{split}\sigma_{Z}^{2}&=1,\\\
\sigma_{B}^{2}&=1-\beta^{2},\\\
\sigma_{A}^{2}&=1-\alpha^{2}-\gamma^{2}-2\alpha\beta\gamma.\end{split}$ (6)

Our model is thus described by the four non-negative parameters $\alpha$,
$\beta$, $\gamma$, and $\delta$, depicted as edge weights in Figure 1, with
the constraint that the quantities in Eq. (6) are non-negative. Those
constraints in turn imply that the parameters are each less than or equal to
1.

Our theoretical results in Theorem 1 and Corollary 1 require understanding the
conditional distributions of model features. For multivariate normal random
variables, these conditional distributions can be computed analytically [40],
allowing us to examine properties of our motivating SEM in more depth.
Specifically, suppose $\mathbf{W}$ is a $k$-dimensional multivariate normal
random variable with mean $\boldsymbol{\mu}$ and covariance $\mathbf{\Sigma}$,
which we partition into into its first $q$ components and its remaining $k-q$
components: $\mathbf{W}=[\mathbf{W_{1}},\mathbf{W_{2}}$]. Further suppose we
accordingly partition $\boldsymbol{\mu}$ and $\mathbf{\Sigma}$ into its
components:

$\displaystyle\boldsymbol{\mu}$
$\displaystyle=\begin{bmatrix}\boldsymbol{\mu_{1}}\\\
\boldsymbol{\mu_{2}}\end{bmatrix}\ \text{with sizes}\begin{bmatrix}q\times
1\\\ (k-q)\times 1\end{bmatrix},$ $\displaystyle\boldsymbol{\Sigma}$
$\displaystyle=\begin{bmatrix}\mathbf{\Sigma_{11}}&\mathbf{\Sigma_{12}}\\\
\mathbf{\Sigma_{21}}&\mathbf{\Sigma_{22}}\end{bmatrix}\ \text{with
sizes}\begin{bmatrix}q\times q&q\times(k-q)\\\ (k-q)\times
q&(k-q)\times(k-q)\end{bmatrix}.$

Then the distribution of $\mathbf{W_{1}}$ conditional on $\mathbf{W_{2}}$ is
multivariate normal with mean

$\boldsymbol{\mu_{1}}+\mathbf{\Sigma_{12}}\mathbf{\Sigma_{22}}^{-1}(\mathbf{W_{2}}-\boldsymbol{\mu_{2}})$

and covariance

$\mathbf{\Sigma_{11}}-\mathbf{\Sigma_{12}}\mathbf{\Sigma_{22}}^{-1}\mathbf{\Sigma_{21}}.$

As a result, the linearity assumption of Corollary 1 is satisfied for
multivariate normal random variables. In particular, in our motivating
example, the conditional distribution of $A_{1}$ given $A_{0}$ and $Z$ is
normal, with

$\displaystyle\mathbb{E}[A_{1}\mid A_{0},Z]$
$\displaystyle=\begin{bmatrix}\sigma_{A_{1}A_{0}}&\sigma_{A_{1}Z}\end{bmatrix}\begin{bmatrix}1&\sigma_{A_{0}Z}\\\
\sigma_{A_{0}Z}&1\end{bmatrix}^{-1}\begin{bmatrix}A_{0}\\\ Z\end{bmatrix}$
$\displaystyle=\frac{1}{1-\sigma_{A_{0}Z}^{2}}\begin{bmatrix}\sigma_{A_{1}A_{0}}&\sigma_{A_{1}Z}\end{bmatrix}\begin{bmatrix}1&-\sigma_{A_{0}Z}\\\
-\sigma_{A_{0}Z}&1\end{bmatrix}\begin{bmatrix}A_{0}\\\ Z\end{bmatrix}$
$\displaystyle=\frac{\sigma_{A_{1}A_{0}}-\sigma_{A_{1}Z}\cdot\sigma_{A_{0}Z}}{1-\sigma_{A_{0}Z}^{2}}A_{0}+\frac{\sigma_{A_{1}Z}-\sigma_{A_{1}A_{0}}\cdot\sigma_{A_{0}Z}}{1-\sigma_{A_{0}Z}^{2}}Z,$

where the $\sigma$ notation denotes the covariance of the indexed random
variables.

Further, the conditional distribution of $(A_{1},Z)$ given $A_{0}$ is likewise
multivariate normal, with covariance matrix

$\displaystyle\begin{bmatrix}1&\sigma_{A_{1}Z}\\\
\sigma_{A_{1}Z}&1\end{bmatrix}-\begin{bmatrix}\sigma_{A_{1}A_{0}}\\\
\sigma_{A_{0}Z}\end{bmatrix}\begin{bmatrix}\sigma_{A_{1}A_{0}}&\sigma_{A_{0}Z}\end{bmatrix}$
$\displaystyle=\begin{bmatrix}1&\sigma_{A_{1}Z}\\\
\sigma_{A_{1}Z}&1\end{bmatrix}-\begin{bmatrix}\sigma_{A_{1}A_{0}}^{2}&\sigma_{A_{1}A_{0}}\cdot\sigma_{A_{0}Z}\\\
\sigma_{A_{1}A_{0}}\cdot\sigma_{A_{0}Z}&\sigma_{A_{0}Z}^{2}\end{bmatrix}$
$\displaystyle=\begin{bmatrix}1-\sigma_{A_{1}A_{0}}^{2}&\sigma_{A_{1}Z}-\sigma_{A_{1}A_{0}}\cdot\sigma_{A_{0}Z}\\\
\sigma_{A_{1}Z}-\sigma_{A_{1}A_{0}}\cdot\sigma_{A_{0}Z}&1-\sigma_{A_{0}Z}^{2}\end{bmatrix}.$

Consequently,

$\mathrm{Cov}(A_{1},Z\mid
A_{0})=\sigma_{A_{1}Z}-\sigma_{A_{1}A_{0}}\cdot\sigma_{A_{0}Z},$ (7)

and, analogously, we have that

$\mathrm{Cov}(B_{1},Z\mid
A_{0})=\sigma_{B_{1}Z}-\sigma_{B_{1}A_{0}}\cdot\sigma_{A_{0}Z}.$ (8)

As above, we can compute the covariances in Eqs. (7) and (8) via the Wright
rules. For example, as seen in Figure 1, there are two $d$-connected paths
between $Z$ and $A_{0}$: the direct connection with edge weight $\alpha$; and
the path through $B_{0}$, with product of edge weights $\beta\gamma$.
Consequently, $\mathrm{Cov}(Z,A_{0})=\alpha+\beta\gamma$. This procedure
allows us to compute all of the terms appearing on the right-hand side of Eqs.
(7) and (8), yielding:

$\displaystyle\begin{split}\sigma_{A_{0}Z}&=\alpha+\beta\gamma\\\
\sigma_{A_{1}Z}&=\alpha+\beta\gamma\\\ \sigma_{B_{1}Z}&=\beta\\\
\sigma_{A_{1}A_{0}}&=\alpha^{2}+2\alpha\beta\gamma+\beta^{2}\gamma^{2}+\gamma^{2}\delta\\\
\sigma_{B_{1}A_{0}}&=\alpha\beta+\beta^{2}\gamma+\gamma\delta.\end{split}$ (9)

Leveraging the above, we now show that $\mathrm{Cov}(A_{1},Z\mid A_{0})\geq
0$, meaning that neighborhood is positively correlated with future arrests,
conditional on past arrests. To see this, first note that

$\displaystyle\delta$ $\displaystyle=\mathrm{Cov}(U_{B_{0}},U_{B_{1}})$
$\displaystyle\leq\sigma_{B}^{2}$ $\displaystyle=1-\beta^{2},$

and so $\beta^{2}+\delta\leq 1$. Now,

$\displaystyle\mathrm{Cov}(A_{1},Z\mid A_{0})$
$\displaystyle=\sigma_{A_{1}Z}-\sigma_{A_{1}A_{0}}\cdot\sigma_{A_{0}Z}$
$\displaystyle=\alpha+\beta\gamma-(\alpha+\beta\gamma)\cdot(\alpha^{2}+2\alpha\beta\gamma+\beta^{2}\gamma^{2}+\gamma^{2}\delta)$
$\displaystyle=(\alpha+\beta\gamma)\cdot(1-\alpha^{2}-2\alpha\beta\gamma-\beta^{2}\gamma^{2}-\gamma^{2}\delta)$
$\displaystyle=(\alpha+\beta\gamma)\cdot(1-\alpha^{2}-2\alpha\beta\gamma-\gamma^{2}(\beta^{2}+\delta))$
$\displaystyle\geq(\alpha+\beta\gamma)\cdot(1-\alpha^{2}-2\alpha\beta\gamma-\gamma^{2})$
$\displaystyle=(\alpha+\beta\gamma)\cdot\sigma_{A}^{2}$ $\displaystyle\geq 0,$

where the first inequality follows from the fact that $\beta^{2}+\delta\leq
1$.

Next we consider $\mathrm{Cov}(B_{1},Z\mid A_{0})$, and note that

$\displaystyle\mathrm{Cov}(B_{1},Z\mid A_{0})$
$\displaystyle=\sigma_{B_{1}Z}-\sigma_{B_{1}A_{0}}\cdot\sigma_{A_{0}Z}$
$\displaystyle=\beta-(\alpha\beta+\beta^{2}\gamma+\gamma\delta)\cdot(\alpha+\beta\gamma).$

In particular, when $\beta=0$, meaning that neighborhood does not impact
behavior, then

$\mathrm{Cov}(B_{1},Z\mid A_{0})=-\alpha\gamma\delta.$

In other words, when neighborhood does not impact behavior (i.e., when
$\beta=0$), neighborhood is negatively correlated with future behavior
conditional on past arrests. (And, by the above, neighborhood is always
positively correlated with future arrests conditional on past arrests.) By
Corollary 1, it is thus better in this case to base predictions of future
behavior solely on past arrests, excluding neighborhood, as we see in Figure
2.