#  RACER: An LLM-powered Methodology for Scalable Analysis of
Semi-structured Mental Health Interviews

Satpreet H Singh1∗ Kevin Jiang 1 Kanchan Bhasin2 Ashutosh Sabharwal2† Nidal
Moukaddam1† Ankit B Patel1,2†

###### Abstract

Semi-structured interviews (SSIs) are a commonly employed data-collection
method in healthcare research, offering in-depth qualitative insights into
subject experiences. Despite their value, the manual analysis of SSIs is
notoriously time-consuming and labor-intensive, in part due to the difficulty
of extracting and categorizing emotional responses, and challenges in scaling
human evaluation for large populations. In this study, we develop RACER, a
Large Language Model (LLM) based expert-guided automated pipeline that
efficiently converts raw interview transcripts into insightful domain-relevant
themes and sub-themes. We used RACER to analyze SSIs conducted with 93
healthcare professionals and trainees to assess the broad personal and
professional mental health impacts of the COVID-19 crisis. RACER achieves
moderately high agreement with two human evaluators (72%), which approaches
the human inter-rater agreement (77%). Interestingly, LLMs and humans struggle
with similar content involving nuanced emotional, ambivalent/dialectical, and
psychological statements. Our study highlights the opportunities and
challenges in using LLMs to improve research efficiency and opens new avenues
for scalable analysis of SSIs in healthcare research.

###### keywords:

Semi-structured interviews , Large language models , Healthcare NLP , Burnout
, COVID-19

00footnotetext:  1Baylor College of Medicine, Houston, TX; 2Rice University,
Houston, TX; ∗ Corresponding author (satpreetsingh@gmail.com); † Equal
contribution

##  1 Introduction

Semi-structured interviews (SSIs) are a widely used qualitative research
method in healthcare research, that provide an in-depth understanding of
subjects’ experiences in their own words [1]. SSIs require interviewers to ask
prespecified ‘root’ questions, along with the option to ask follow-up
questions to gain clarity on the interviewee’s responses. This flexibility is
a key characteristic of SSIs, allowing for a more dynamic and responsive data
collection process, especially in areas where exploratory forays are needed.
The adaptability of SSIs is particularly beneficial in exploring complex or
sensitive topics such as mental health. SSIs allow rapport building between
interviewer and subject, and facilitate candid responses on sensitive matters.
The open-ended nature of follow-up questions gives subjects the freedom to
reflect on experiences and articulate thoughts without judgement. This helps
reveal the nuances, contradictions, and diversity of perspectives that
traditional fixed quantitative surveys may overlook. However, the traditional
manual analysis of these interviews is a time-consuming and resource-intensive
process. The advent of Large Language Models (LLMs), such as GPT-4 [2, 3, 4],
offers a novel and efficient method to extract and interpret data from such
text corpora. Yet, the validity of LLMs in analyzing emotional states may be
limited in circumstances where participants express multiple emotions or
conflicting (dialectical) states.

As a case-study, we leveraged data from SSIs, conducted during the peak of the
COVID-19 crisis in 2020, to understand the mental well-being of 93 healthcare
professionals and trainees. The COVID-19 pandemic brought to the forefront
profound personal and professional challenges experienced by healthcare
workers. Fear of infecting family members, grief over patient deaths, moral
dilemmas in resource allocation, and anxieties about professional preparedness
collectively introduced a heightened level of psychological complexity and
stress in the lives of healthcare professionals. The stigma surrounding the
pursuit of mental health support exacerbated these challenges, leaving
healthcare workers hesitant to openly discuss their difficulties or seek
assistance.

In this paper, we developed RACER, an expert-guided automated pipeline that
Retrieved responses to about 40 questions per SSI, Aggregated responses to
each question across all subjects, Clustered these responses for each question
into insightful domain-relevant Expert-guided themes [5], and finally Re-
clustered responses to produce a robust result. Human evaluation on a subset
of the total population revealed moderately high agreement [6] between humans
and RACER outputs, and similarities between inter-human disagreement and
human-machine disagreement. We summarize our findings from applying RACER to
our SSI-survey on the experiences of healthcare professionals and trainees
during COVID-19, to reveal the power of this approach. Our results demonstrate
both the capabilities and the limitations leveraging LLMs to efficiently
process and extract insights from a large corpus of SSIs.

##  2 Results

Table 1: Demographic Characteristics of the Study Population Characteristic | Percentage
---|---
Gender |
Male | 54.84%
Female | 45.16%
Age Group |
22-33 years | 39.78%
34-45 years | 32.26%
46-60 years | 16.13%
61+ years | 5.38%
Unclear/Excluded | 6.45%
Healthcare Professional/Student Type |
Physicians | 54.84%
Medical Students | 21.51%
Nurses | 8.60%
Residents | 7.53%
Other Professionals | 12.90%
Unclear/Excluded | 1.08%
Location |
Houston, Texas | 44.09%
Other Texas | 21.50%
Florida | 10.75%
Mid-West US | 13.98%
Other US | 5.38%
Unclear/Excluded | 2.15%
Marital Status |
Not married | 41.94%
Married | 52.69%
Unclear/Excluded | 5.38%
Have Kids? |
Yes | 51.61%
No | 45.16%
Unclear/Excluded | 3.23%
Specialty Area |
Emergency Medicine | 26.88%
Psychiatry | 16.13%
Pulmonary Critical Care | 16.13%
Internal Medicine | 11.83%
Neurology/Neurocritical Care | 5.38%
Surgery/ER | 5.38%
Pediatrics | 5.38%
Other Specialties | 17.22%
Unclear/Excluded | 2.15%
Years of Practice |
Under 15 Years | 71.23%
15-30 Years | 20.55%
Over 30 Years | 5.48%
Unclear/Excluded | 1.37%
Figure 1:  Stages of the RACER (Retrieve, Aggregate, Cluster with Expert
guidance, and Re-cluster) Semi-Structured Interview (SSI) processing pipeline:
First, Retrieve relevant responses to each SSI question. Aggregate responses
across subjects before Clustering them into themes (and subthemes) defined by
Experts. To assess robustness, Re-cluster multiple times and make assignments
by majority vote. The pipeline efficiently converts SSI text into meaningful
themes with confidence scores.

###  2.1 Recruitment and interview of a diverse sample of healthcare
professionals and trainees

Healthcare professionals and trainees across different specialties and career
stages were recruited via snowball sampling method [7], described as follows.
The investigators asked colleagues if they knew of anyone willing to
participate in interviews about their COVID-19 experiences. Announcements were
also posted online and through professional networks. Participation was
voluntary with no compensation provided. Approval was obtained from the Baylor
College of Medicine (Houston, TX) Institutional Review Board. The interviews
were performed by a team of two research coordinators with healthcare
backgrounds, and a third-year medical student, under the supervision of the
investigators.

The study population of healthcare professionals and trainees consisted of 93
subjects (51 male, 42 female) with diverse demographics (Table 1). Subjects
were from 22 years to over 61 years in age, and were located predominantly in
Texas. Over half were married and had children. Most (75%) had no care-taking
responsibilities in addition to child-care. Professionally, the sample
included physicians (54.8%), medical students (21.5%), nurses (8.6%),
residents (7.5%) and other healthcare professionals. Subjects trained at
multiple institutions, with prominent representation from Baylor College of
Medicine and University of Texas systems. Various specialties were represented
in the cohort, with emergency medicine, psychiatry and pulmonary/critical care
among the most common.

SSIs were conducted over videoconferencing using a standard template
consisting of a total of 41 questions, including four questions that were only
asked to students, and seven questions that were asked to only non-students.
Questions were either factual, concerning demographics and personal and
professional background, or open ended, where interviewees were asked to talk
about their experiences during the COVID-19 pandemic, focusing on their
exposure to the virus, work impacts, emotional responses, future outlooks, and
coping strategies. Interviewees discussed how they had practiced in high-risk
areas, their concerns for personal and family safety, and modifications made
to their routines. They also reflected on the physical toll the crisis had
taken. The impact on their work included changes in working hours, shifts in
patient care quality, and altered management approaches. Emotional and
psychological questions revealed how the crisis affected them emotionally, the
level of support they received, family dynamics, and changes in burnout
levels. Looking ahead, they pondered the crisis’s short-term and long-term
impacts on their careers and specialty choices. Finally, they shared their
openness to seeking help for burnout or mental overwhelm and identified
potential obstacles in obtaining this help. Students were not asked clinical-
practice related questions, and were instead asked about how their training
was being affected by pandemic-related changes. Interviews lasted on average
26.7 +/- 8.9 s.d. minutes. When transcribed from raw interview audio into text
transcripts (using Otter.AI[8]), were on average 4044.30 +/- 1348.34 s.d.
words long.

Figure 2:  Human-RACER approaches resembles human-human disagreement: (A)
Transcript segments from two different subjects being asked “How the
[COVID-19] crisis has affected you emotionally” that were either all
concordant or all non-concordant between evaluators, displaying the ambiguity
that exists in parsing free response. (B) The concordance ratio calculated
between evaluator pairs and between all evaluators. A chi-squared test with
yates continuity correction between the three different evaluator pairings
showed human evaluator concordance did not differ from evaluator one’s
concordance with RACER. * p $<$ 0.5, ** p $<$ 0.01, *** p $<$ 0.001, **** p
$<$ 0.0001.

###  2.2 RACER extracts relevant interviewee responses and robustly clusters
them

We developed an LLM-based automated pipeline called RACER (Figure 1) that
converts a corpus of text SSI transcripts into insightful themes per interview
question. RACER, consists of four stages, Retrieve, Aggregate, Cluster with
Expert guidance, and Recluster:

_Retrieve_ : We first structured interview transcripts by using an LLM
(OpenAI’s GPT-4[2]) to retrieve relevant SSI text in response to each of the
questions in the interview template. (See Methods LLM prompt details) To avoid
LLM ‘hallucinations’ [9], we asked the LLM to provide ‘evidence’ in the form
of text quoted verbatim from the transcript, to back up its response to each
question. LLM outputs missing either answers or backing evidence to any
question were automatically detected and rerun.

_Aggregate_ : For each question, we then aggregated the retrieved responses
across all subjects who were asked that question.

_Cluster with Expert guidance_ : We then asked the LLM to semantically cluster
the responses into primary and secondary clusters (‘themes’ and ‘sub-themes’).
For most questions, we provided the LLM expert-guidance in the form of
primary-cluster definitions. The LLM discovered secondary clusters (or sub-
themes) automatically. Expert-provided cluster definitions were always
mutually exclusive and collectively exhaustive, while those discovered by the
LLM were not constrained to be so. Similar to before, invalid LLM responses,
e.g. those missing cluster assignments for any subjects, were automatically
re-run.

_Re-Cluster_ : Leveraging the probabilistic nature of LLMs, we assessed the
robustness of the clustering process by re-running it four more times,
employing the same cluster definitions and validation criteria as in the
initial step. We used a majority vote over 5 runs to assign subjects to
clusters, to get robust cluster assignments for all downstream processing. The
number of votes (3, 4 or 5 out of total 5 LLM calls) additionally provided a
synthetic measure of LLM confidence [10, 11]. Only a very small fraction of
subject-question pairs (12 out of 3342, 0.36%) had no ‘robust’ cluster
assignments after applying the majority voting process.

All together, we found that RACER was able to take unstructured transcriptions
and extract relevant and insightful, clustered responses in a robust manner
for downstream human analysis.

###  2.3  Human-machine disagreement approaches inter-human disagreement

To validate the output of running RACER on our SSI dataset, two human
evaluators cross-checked the resulting cluster assignments for 20 randomly-
selected subjects across 28 open ended questions. Using the same cluster
definitions as were previously used by RACER, each human evaluator (E1 and E2)
independently read the raw transcript file and assigned each subject’s answers
to the primary clusters. Evaluator cluster assignments were then compared to
RACER’s robust cluster assignments. To quantify agreement, we defined a
concordance score and a concordance ratio as follows: If the clusters for a
given subject-question pair matched exactly (for mutually exclusive clusters),
or were sub-sets or super-sets of each other (for mutually non-exclusive
clusters) they were assigned a concordance score of 1. Conversely, mismatch
was assigned a concordance score of 0. The overall concordance ratio is the
proportion of matched subject-question pairs between evaluators.

We observed a concordance of 78% (E1) and 87% (E2) between human evaluators
and RACER, and a 77% (E1-E2) inter-rater concordance (Figure 2B). When all
three evaluators were compared simultaneously, there was a little decrease in
the concordance (72%), indicating that across the majority of subject-question
pairings, cluster assignments produced by humans and RACER were all in
agreement. See Methods for additional details.

###  2.4 Machine ”confusion” resembles human confusion

Figure 3:  RACER “confidence” correlates with evaluator concordance and
reveals areas of human disagreement: (A) Distribution of the proportion of
subject-question pairs that were clustered robustly across subject-question
pairs human evaluators examined (20 subjects) or across all subject question
pairs (93 subjects). (B) Average RACER confidence scores for all subjects (n =
93) for a given question correlate significantly with the evaluator pair
concordance (n = 20) using Spearman Rank. (C) Average RACER confidence scores
calculated within concordant vs non-concordant subject-question pairs between
evaluators. Chi-square test was conducted to determine if distribution of
confidence scores differed between concordant vs non-concordant subject-
question pairs.Correlation. * p $<$ 0.5, ** p $<$ 0.01, *** p $<$ 0.001, ****
p $<$ 0.0001.

We examined the confidence score produced by RACER per subject-question pair
to see how it might affect the subject-question pair’s concordance with human
evaluators (Figure 3). Amongst the 443 subject-question pair sample evaluated
by humans, 392 (87.7%) were entirely robust (5 of 5 repeat clusters). This was
similar to the population confidence score distribution 88.2% (1852 of 2099
subject-question pairs). RACER’s average confidence across all subjects for a
given question showed significant and positive correlations with Human-RACER
concordance of the 20 subjects evaluated for those same questions.
Additionally, we observed that the confidence scores of subject-question pairs
that were concordant between human evaluators and RACER, were higher than for
non-concordant subject-question pairs. This was due to significant differences
in the proportion of lower confidence subject-question pairs between
concordant and non-concordant groups. Interestingly, when we juxtaposed RACER
confidence scores against human-human concordance, we observed that RACER
confidence was lower when humans were non-concordant. This suggests that areas
where RACER was less confident or ‘confused’ were also areas where human
evaluators tended to disagree. Thus the RACER confidence generated via
repeated clustering could also serve as an indicator of SSI ambiguity or
difficulty.

###  2.5 Insights using RACER on healthcare worker experience during COVID-19

We now summarize the insights gleaned from analyzing SSIs with 93 subjects
using our automated processing pipeline.

####  2.5.1 COVID-19 exposure, response, work impact and work changes

(A)

(B)

(C)

(D)

(E)

(F)

Figure 4: Aggregated interview responses to selected questions about safety
concerns arising from COVID-19 exposure, work impact, and medical management
decisions. Error bars reflect cluster-assignment variability arising from re-
clustering step in RACER.

The vast majority of practicing healthcare professionals reported having
professional contact with COVID-19 patients in the past two months. Most
subjects expressed safety concerns for themselves and loved ones, especially
regarding viral exposure risks. Common protective measures adopted included
heightened hygiene practices, using personal protective equipment, limiting
travel and social interactions, and modifying routines at work and home to
minimize transmission risks. Over half of the subjects reported physical tolls
from the crisis, frequently citing exhaustion, disturbed sleep, and dietary
changes (Figure 4).

Most subjects felt personally prepared to handle the pandemic, attributing
this largely to their medical knowledge, experience, and ability to adapt.
Assessment of institutional preparedness was more varied, with around 60%
expressing their hospital/unit was prepared, but around 25% felt improvements
were still needed.

Working hours markedly increased for most subjects during the pandemic, with
over 80% reporting working more than 40 hours per week compared to pre-COVID
times. For many, this resulted from escalations in patient load and
administrative duties. Approaches to patient management also evolved, with the
vast majority of practicing healthcare professionals stating their methods
differed from usual practices. This included increased reliance on technology,
more precautions with patients, and adjustments to treatments due to COVID-19.
Most still felt capable of handling the situation professionally, though some
desired more protections and support systems.

Among students and trainees, the majority believed they adhered closely to the
Hippocratic oath during the pandemic. Their views on their educational
institution’s policies regarding medical students’ roles during that time were
divided, with half in agreement and others expressing mixed or negative
sentiments, reflecting a spectrum of perspectives on the adequacy and
effectiveness of institutional responses to the crisis.

####  2.5.2 Emotional and psychological impact, and support and coping
strategies

(A)

(B)

(C)

(D)

(E)

(F)

Figure 5:  Aggregated interview responses to selected questions about
emotional and psychological impact, and support and coping strategies. Error
bars reflect cluster-assignment variability arising from re-clustering step in
RACER.

The COVID-19 crisis negatively affected the emotional state of most subjects,
with many reporting feelings of anxiety, stress, sadness, or anger. However,
around 25% indicated a mix of both positive emotions like gratitude as well as
negative feelings. Despite those challenges, the overwhelming majority felt
supported by peers and family, suggesting strong social networks within and
outside the workplace. Family dynamics had been affected for some, with around
a quarter reporting increased family problems during the pandemic. This data
underscored the profound emotional and psychological effects of the crisis on
healthcare professionals, juxtaposed with the resilience and support systems
that helped them navigate these challenges.

In regards to burnout, over 60% of subjects assessed their pre-pandemic
burnout as low or mild. When asked about current burnout, around 40% still
reported mild or no burnout, but the percentage reporting severe burnout rose
from around 15% pre-pandemic to 20% during the crisis. If feeling burned out,
nearly 90% stated they would seek help, with most mentioning professional
resources like counseling. Over 60% also reported they would seek professional
help if feeling mentally overwhelmed, with therapists and workplace programs
being commonly cited options. However, around 45% still anticipated obstacles
in getting help, including logistical barriers and stigma concerns (Figure 5).

####  2.5.3 Future considerations and professional outlook

(A)

(B)

(C)

(D)

Figure 6: Aggregated interview responses to selected questions about future
considerations and professional outlook, as it relates to working in
healthcare during or after the pandemic. Error bars reflect cluster-assignment
variability arising from re-clustering step in RACER.

When asked about near-term impacts, over 50% expressed concerns about
anticipated difficulties, health risks, economic instability, and significant
lifestyle changes. However, around 15% hoped for new opportunities and growth
resulting from the crisis. Looking 5 years ahead, around 20% expected
advancements in healthcare practices and systems due to learned lessons.
Though nearly 10% feared lingering personal and professional impact. Among
non-students considering job changes, around 15% expressed an immediate
willingness to switch fields while around 18% would change contingent on
worsening conditions.

Regarding effects on career plans, 35% of students reported the crisis has
impacted their specialty choices or work preferences. Specifically, around 20%
described reconsidering their specialty choice due to the pandemic. Another
15% mentioned shifting their preferences regarding research involvement,
practice locations, and other factors. However, 50% of students stated the
crisis has not affected their professional plans or specialty decisions. Over
50% of students explicitly stated adherence to their Hippocratic oath
obligations, while 10% conveyed adherence through descriptions of their
clinical actions and interventions. Of students agreeing with their school’s
pandemic policies, 40% expressed unqualified agreement and 10% provided
positive justifications. However, around 15% agreed tentatively due to
concerns over student safety and curriculum changes (Figure 6).

##  3 Discussion

###  3.1 Summary

Our study demonstrates the utility of RACER for efficiently analyzing semi-
structured interviews (SSIs), particularly those exploring complex mental
health topics within the healthcare domain. We introduce a novel approach by
employing RACER to analyze emotions and psychological behaviors, opening new
possibilities for exploration in mental health. By providing expert-guided
constraints and using automated response validation steps, RACER accurately
extracts and robustly clusters relevant responses from interview transcripts.
Automating these laborious manual tasks significantly enhances the scalability
of SSI analysis. The inter-rater agreement between LLM-assigned clusters and
human expert clusters further bolsters our claims. The automated pipeline
achieved moderately high concordance compared with manual evaluation by human
annotators. The overall concordance ratio of 0.72 for RACER versus both human
evaluators approaches the 0.77 concordance ratio between the two human
evaluators.

###  3.2 Limitations and tradeoffs

Our findings reveal both the promises and current pitfalls of LLMs for SSI
analysis. We found that when the RACER struggled with robust clustering, both
humans and machine were more likely to be non-concordant which could suggest
shared limitations in handling complex emotions or psychologically nuanced
statements [12] or ambiguity of the underlying SSI. This underscores the
indispensable role of human expertise in reviewing and interpreting LLM
outputs, where RACER’s confidence levels can guide expert scrutiny.

While RACER provided evidence in the form of quoting relevant interview text
to support its response in the Retrieval step, the underlying methodology
remains opaque. In contrast, human evaluators were able to describe their
techniques, even if subjective. For instance, humans considered different
amounts of contextual information outside the question scope, and inferred
subject intentions to varying degrees, i.e. whether the subject needed to
explicitly say certain phrases, or if they could be inferred from previous
statements or knowledge of the subject matter. An LLM’s ability to consider
large amount of contextual information can be a double-edged sword; beneficial
if relevant information appears elsewhere in the transcript, but misleading if
the research is indeed directed towards a narrow window of text around the
question.

We demonstrated that LLMs can help discover knowledge by automatically
extracting themes and topics from subject responses. However, good performance
requires clear, mutually exclusive category definitions. We found it highly
useful to involve domain experts early to precisely define mutually exclusive
thematic clusters. For certain questions, where succinct mutually exclusive
categorization was not possible, we chose to use LLM-discovered clusters.
However, validation of such non-exclusive categorization is challenging. Our
results showed higher LLM accuracy and inter-rater agreement for questions
with non-overlapping expert-defined clusters versus those allowing multiple
clusters.

Additionally, human evaluators exhibited biases, such as default cluster
tendencies requiring countering evidence (e.g. starting from a default of ‘no’
and requiring evidence to switch to a ‘yes’, or vice versa). Thus, expert
human analysis also demonstrates cognitive variability and individual biases.
Rather than definitive classifications, both human and machine outputs should
be considered informed yet inherently biased perspectives on complex
qualitative responses [13]. Thus, in the future, clearly delineating the
parameters of evaluations with humans and RACER may improve concordance.

While RACER’s cluster assignments may deviate slightly from human reviewers,
RACER was internally consistent and demonstrated high clustering repeatability
for most questions. Furthermore, unlike humans, RACER was able to efficiently
process an extensive dataset of 93 subjects and can scale to significantly
larger data set sizes that would otherwise be infeasible for human evaluators
to handle.

###  3.3 Future work

For researchers undertaking projects in this emerging domain, both optimism
and caution are warranted [14, 15, 16, 17, 18, 19]. With appropriate
constraints and validation, LLMs can accelerate knowledge extraction from
SSIs. We implemented safeguards against hallucination risks like requiring
verbatim textual evidence for an answer, which constrained the LLM to mostly
avoid fabricating content. While this is already an area of active research,
the possibility of a few false positives remains and needs to be accounted for
in downstream use.

While evaluation of LLM outputs through comparison to multiple human raters
was helpful through comparison to multiple human raters, inter-rater agreement
must also be looked at to assess inherent ambiguity. To further improve
performance, we recommend specialized training for both SSI interviewers and
human evaluators.

We found it useful to generate an ensemble of LLM clustering outputs from
repeated runs, and used it to extract robust cluster assignments and to get a
measure of model uncertainty. Future work exploring this direction could
produce useful methods that help build trust in LLM-assisted analyses and
inform human-in-the-loop processes for high-stakes applications [20].

##  4 Methods

###  4.1 Semi-structured interviews

Interviewers were provided with a standard template to guide their
discussions. The subjects were all healthcare professionals or trainees,
including physicians, nurses, and medical students. The interviews followed a
semi-structured format, where the interviewers were instructed to cover a
previously decided list of questions, and were allowed to ask exploration
questions if the ‘root’ question was not answered. The questions covered in
the SSIs are listed in A. Raw audio / video interview files were transcribed
into text format using the Otter.AI transcription service [8]. Out of 100
interviews conducted, 7 were compromised due to data-corruption/loss issues,
providing a total of 93 transcriptions for further processing. Voice to text
transcription was carried out using Otter.AI[8], which attempts to perform
automated speaker diarization, but does not do so perfectly. To the best of
our knowledge, this shortcoming did not seem to influence the subsequent
processing steps.

###  4.2 RACER

We used the OpenAI GPT-4 LLM for all our work, except for prompts which
exceeded GPT-4’s limits, where we used GPT-4-32k.

####  4.2.1 Retrieval

In this step, the model was tasked with retrieving relevant responses for each
question from a predefined list of questions (listed in A) from the
transcript. The prompt for the LLM consisted of instructions and a template
consisting of the aforementioned list of questions and what format each
question’s response should be in, followed by the entire SSI transcript. The
full prompt is detailed in A.

LLM Response Validation: By asking the LLM to respond in a structured format,
we could partially automate the process of verifying the LLM’s response. The
LLM is called once for each subject, and then the response is parsed using the
Python Pandas library. The LLM’s response is marked invalid if it is ill-
formatted (not parsable in tab-separated-values format) or incomplete (wrong
number of rows, i.e. questions, or columns, i.e. incomplete response). The LLM
is called again on invalid responses till the LLM returns a valid response. We
found that at most 4-5 (5%-6%) subjects would have invalid responses in the
first attempt, and in total, we were making about 10% additional calls to get
valid responses for all subjects. The most common issues were that the LLM
would sometimes be incomplete (skip questions, end output before final
question) and sometimes use the specified tab-delimiter incorrectly.

####  4.2.2 Cluster with Expert guidance

In this step, we employed a semantic clustering approach which grouped
responses based on the underlying themes or sentiments (”semantic clusters”)
they conveyed. In preliminary explorations, we found that the LLM is able to
automatically generate interesting semantic clusters from a list of the
subjects’ responses without additional human guidance. However, in many cases
(21 out of $\approx$40 questions), we felt like it was important to exercise
more control over the LLM’s response to facilitate human consumption of the
results. So, we provided expert guidance in the form of a list of primary
clusters or ”themes” (defined on a per-question basis), which were included in
the prompt using a template (detailed in B). Secondary clusters or ”sub-
themes” were discovered automatically by the LLM. Each subject’s response was
mapped exclusively to one primary cluster and could furthermore be associated
with one or more secondary clusters.

LLM Response Validation: The LLM returned two lists in its response: one of
the cluster labels and their definitions, and the other of the cluster-labels
(single or two-level clustering) assigned to each subject. The LLM was called
once for each of 40 questions, and these responses were parsed using the
Python Pandas library. A LLM response is marked invalid if it was ill-
formatted or incomplete. The LLM was called again on invalid responses till
the LLM returned a valid response. We found that almost 20 questions would
have invalid responses in the first attempt, and in total, we were making
almost 80% additional calls to get valid responses for all questions. We
suspect that the rate of invalid responses in this step is higher than in the
previous step due to the added complexity of the task i.e. the response needs
to first produce a valid clustering-schema, and then additionally assign each
of 93 subjects to the clusters according to the clustering schema.

####  4.2.3 Recluster

We repeated the above clustering step four additional times using a similar
prompt (detailed in C) with previously defined cluster-definitions. As in the
original clustering, any invalid LLM responses were automatically detected and
re-processed until a valid response was obtained. For the final cluster
assignments used in downstream analysis, we applied a majority vote rule based
on the 5 clustering repetitions. In a few cases ($<1\%$ of all subject-
question pairs), this process failed to find any cluster assignments that
passed the majority-vote.

###  4.3 Human evaluation of LLM responses

Our study employed human evaluation to verify the alignment between RACER-
generated clusters and human interpretation, utilizing two independent
evaluators who analyzed the responses of 20 randomly selected subjects from a
pool of 93. Each evaluator individually reviewed the raw interview transcript
files for the selected 20 subjects and used the same cluster definitions as
RACER to assign subjects to clusters. Human evaluators spent approximately 30
minutes per subject on average for a comprehensive review and categorization
of the responses. This time investment reflects the thoroughness and attention
to detail applied by the evaluators in their analysis, and also highlights the
limits of this process to scale to large study populations. To validate the
semantic clustering results produced by the LLM, each human evaluator compared
their assigned scores with those generated by the LLM. An inter-rater
comparison was also conducted, involving a detailed examination of the scores
and evaluations independently made by both human evaluators (E1 and E2) for
the same set of subjects. Concordance scores of 1 were assigned to clusters
that precisely matched or were sub- or super-sets of each other, while
discrepancies received a concordance score of 0. The overall concordance ratio
represented the proportion of clusters aligning between the evaluators.

Additionally, the evaluators’ findings were juxtaposed with RACER’s cluster
assignments to gauge both inter-evaluator consistency and the degree of
correspondence with the LLM’s outcomes. We also compared the use of Cohen’s
kappa coefficient with our concordance score and found them to be similar
(Appendix 8). Due to the nature of the comparison across questions which
varied in the number of possible clusters as well as probability of different
cluster assignment across questions, the concordance scores were used as they
better described the intended comparisons. Instances where RACER did not
produce any robust cluster assignments were categorized as ’mismatch’ during
the evaluation process.

## Acknowledgements

We thank the involved research staff and consenting subjects for supporting
this research.

## Author Contributions Statement

SHS, NM, AS and AP conceived of the study/analysis. NM supervised the
collection of interviews. SHS designed and coded the automated interview
transcription, LLM processing, data-analysis, and visualization pipelines. SHS
and NM evaluated early LLM performance and designed the expert-guided
clustering criteria. KJ and KB performed the human evaluation, data
interpretation and writing related to human evaluation. KJ visualized and
analyzed the human evaluation data. AP funded the transcription and large-
language model accounts. SHS wrote the first draft of the manuscript,
excluding text related to human evaluation. All authors interpreted the
results and reviewed and edited the manuscript.

## Competing Interests Statement

All authors declare no financial or non-financial competing interests.

## Code Availability Statement

Code used to generate our results can be found at
https://github.com/satpreetsingh/RACER

## Data Availability Statement

Aggregated cluster assignment data required to reproduce figures has been made
publicly available on the accompanying code repository. Human evaluation
statistical analyses and visualizations were performed with Prism 10 (GraphPad
Software). Data from earlier stages of the pipeline contain Personally
identifiable information (PII) and therefore has not been released.

## References

  * [1] Eike Adams.  The joys and challenges of semi-structured interviewing.  Community Practitioner, 83(7):18–22, 2010.
  * [2] Peter Lee, Carey Goldberg, and Isaac Kohane.  The AI revolution in medicine: GPT-4 and beyond.  Pearson, 2023.
  * [3] Peter Lee, Sebastien Bubeck, and Joseph Petro.  Benefits, limits, and risks of gpt-4 as an ai chatbot for medicine.  New England Journal of Medicine, 388(13):1233–1239, 2023.
  * [4] V Vien Lee, Stephanie CC van der Lubbe, Lay Hoon Goh, and Jose M Valderas.  Harnessing chatgpt for thematic analysis: Are we ready?  arXiv preprint arXiv:2310.14545, 2023.
  * [5] V. Vien Lee, Stephanie C. C. van der Lubbe, Lay Hoon Goh, and Jose M. Valderas.  Harnessing ChatGPT for thematic analysis: Are we ready?, October 2023.  arXiv:2310.14545 [cs].
  * [6] Mary L McHugh.  Interrater reliability: the kappa statistic.  Biochemia medica, 22(3):276–282, 2012.
  * [7] Leo A Goodman.  Snowball sampling.  The annals of mathematical statistics, pages 148–170, 1961.
  * [8] Otter.ai.  Otter.ai: Voice meeting notes transcription service, 2023.  Accessed: 2023-12-10.
  * [9] SM Tonmoy, SM Zaman, Vinija Jain, Anku Rani, Vipula Rawte, Aman Chadha, and Amitava Das.  A comprehensive survey of hallucination mitigation techniques in large language models.  arXiv preprint arXiv:2401.01313, 2024.
  * [10] Benjamin Kompa, Jasper Snoek, and Andrew L Beam.  Second opinion needed: communicating uncertainty in medical machine learning.  NPJ Digital Medicine, 4(1):4, 2021.
  * [11] Sree Harsha Tanneru, Chirag Agarwal, and Himabindu Lakkaraju.  Quantifying uncertainty in natural language explanations of large language models.  arXiv preprint arXiv:2311.03533, 2023.
  * [12] William Boag, Olga Kovaleva, Thomas H. McCoy, Anna Rumshisky, Peter Szolovits, and Roy H. Perlis.  Hard for humans, hard for machines: predicting readmission after psychiatric hospitalization using narrative notes.  Translational Psychiatry, 11(1):1–6, January 2021.  Number: 1 Publisher: Nature Publishing Group.
  * [13] Mohammad Atari, Mona J. Xue, Peter S. Park, Damián Blasi, and Joseph Henrich.  Which Humans?  January 2024.  Publisher: OSF.
  * [14] Kimberly Badal, Carmen M. Lee, and Laura J. Esserman.  Guiding principles for the responsible development of artificial intelligence tools for healthcare.  Communications Medicine, 3(1):47, April 2023.
  * [15] Debadutta Dash, Rahul Thapa, Juan M Banda, Akshay Swaminathan, Mehr Kashyap, Nikesh Kotecha, Jonathan H Chen, Saurabh Gombar, Lance Downing, Rachel Pedreira, Ethan Goh, Angel Arnaout, Garret K Morris, Matthew P Lungren, Eric Horvitz, and Nigam H Shah.  Evaluation of GPT-3.5 and GPT-4 for supporting real-world information needs in healthcare delivery.
  * [16] Yu Ying Chiu, Ashish Sharma, Inna Wanyin Lin, and Tim Althoff.  A Computational Framework for Behavioral Assessment of LLM Therapists, January 2024.  arXiv:2401.00820 [cs].
  * [17] Liyan Tang, Zhaoyi Sun, Betina Idnay, Jordan G. Nestor, Ali Soroush, Pierre A. Elias, Ziyang Xu, Ying Ding, Greg Durrett, Justin F. Rousseau, Chunhua Weng, and Yifan Peng.  Evaluating large language models on medical evidence summarization.  npj Digital Medicine, 6(1):158, August 2023.
  * [18] Michael Wornow, Yizhe Xu, Rahul Thapa, Birju Patel, Ethan Steinberg, Scott Fleming, Michael A. Pfeffer, Jason Fries, and Nigam H. Shah.  The shaky foundations of large language models and foundation models for electronic health records.  npj Digital Medicine, 6(1):135, July 2023.
  * [19] Nigam H. Shah, David Entwistle, and Michael A. Pfeffer.  Creation and Adoption of Large Language Models in Medicine.  JAMA, 330(9):866, September 2023.
  * [20] Nadine Bienefeld, Jens Michael Boss, Rahel Lüthy, Dominique Brodbeck, Jan Azzati, Mirco Blaser, Jan Willms, and Emanuela Keller.  Solving the explainable AI conundrum by bridging clinicians’ needs and developers’ goals.  npj Digital Medicine, 6(1):94, May 2023.

APPENDICES

##  Appendix A LLM prompt for retrieving relevant responses from interview
transcripts

⬇

Here is a template (tab-separated-values) of an interview (conducted in 2020)
between an interviewer and a healthcare professional or medical student.

Populate the ’answer’ column of the template below using the interview
transcript appended after the template.

Be sure to note any positive, negative or neutral emotions expressed by the
interviewee in the answer.

If a template question was not asked in the appended transcript (or is not
applicable), the answer should be "NA".

For the last ’evidence’ column, provide evidence, by quoting verbatim (except
for newlines) the parts of the transcript that were most relevant to answering
the question.

question_number question answer evidence

1 How old are you? [numeric]

2 Where do you live? [city, state, country]

3 What is your marital status? [single/married/divorced/widowed/etc]

4 Do you have kids? [yes/no]

5 If you do have kids, provide details [details]

6 Are you a caretaker otherwise? (if not own kids, eg elderly parents, adopted
family member, etc) [yes/no; details]

7 What type of healthcare professional or student/trainee are you? [details]

8 If student or trainee, what year are you in? [year of program]

9 What institution did you complete your (or are currently) training at? [name
and location of institution]

10 If you are a physician, did you train in the US at any point? [yes/no]

11 What is your specialty (if student, what specialty are you thinking of)?
[details]

12 How long have you been practicing? [in years, or NA for student]

13 Over the past two months, have you practiced clinically in areas where you
could be in touch with patients who have covid-19? [yes/no]

14 Are you concerned about your safety, and how? [yes/no; details]

15 Are you concerned about safety of loved ones, and how? [yes/no; details]

16 Have you modified your routine to protect yourself or others, and how?
[yes/no; details]

17 Has this crisis taken a toll on you physically in any way? [yes/no;
details]

18 How many hours are you working on average (per week) nowadays? [numeric]

19 How has your working schedule and logistics changed? [details]

20 How do your working hours compare to pre-covid-19 crisis? [details]

21 How do think the covid-19 crisis has affected the quality of patient care?
[details]

22 How has it changed your approach to management? (different from usual, at
odds with existing guidelines, may not be as effective, etc.) [details]

23 Are your processes different for end-of-life decisions? Do you have to take
people off ventilator more frequently? [details]

24 How prepared do you feel for the COVID-19 pandemic on a personal level?
[details]

25 How prepared do you feel the unit/hospital is for the COVID-19 pandemic?
[details]

26 How do you think this crisis has affected you emotionally? [note emotions
recognized from interviewee;details]

27 Do you feel supported by peers and/or family during this time? [details]

28 Have you had more problems with family during this time? [details]

29 Before this crisis, if someone asked you about your burnout level, what
would you have answered? [score (e.g. 6 out of 10) and/or details]

30 How burned out do you feel nowadays (during the ongoing COVID crisis)?
[score (e.g. 6 out of 10) and/or details]

31 How do you feel about working from home OR at the frontlines?
[Home/Frontlines/Other; details]

32 Do you feel you should be able to handle this as a healthcare professional?
[yes/no; details]

33 What impact do you see this crisis having on you in the near future?
[details]

34 What impact do you see this crisis having on you about five years from now?
[details]

35 Would you seek help if you felt burned out? How? [yes/no; details]

36 Would you change jobs or career trajectories? [yes/no; details]

37 Has this crisis affected your specialty decision or career plans in any
way? [yes/no; details]

38 Would you get (professional or other) help/care if you felt mentally
overwhelmed? How? When? [yes/no; details]

39 Any obstacles you foresee in getting help if you needed to? [yes/no;
details]

40 If student or trainee, how closely do you feel that you are adhering to the
Hippocratic oath during this time? [closely/not-closely; details]

41 If student or trainee, do you agree with your school’s policies regarding
medical students’ roles at this time? [yes/no; details]

TRANSCRIPT:

[Interview Transcript Appended]

##  Appendix B LLM prompt template for semantic Clustering of responses
aggregated across all subjects

Out of 40 questions in our template in A, 21 questions had expert-provided
templates that defined the primary clusters but left secondary-cluster
definitions to the LLM. 4 questions (Q13, Q15, Q17, Q19) used LLM-discovered
single-level clustering. The following Python code shows the template used for
generating the prompt associated with each question:

⬇

TEMPLATE = """Cluster the responses in the table below at two levels.

Top level clusters must be {clusters}.

Top level clusters have mutually-exclusive cluster membership.

For the next level, cluster the responses from subjects belonging to each top-
level cluster highlighting the common theme per cluster.

Subjects can belong to multiple clusters at this level.

Your response should be in tab-separated-values format, with the following
columns:

subject_id top_level_cluster_id secondary_cluster_ids

Example output line:

C-002 C1 "C1.1,C1.2,C1.4"

Start your response by defining each top and secondary cluster in tab-
separated-values format, with columns:

cluster_id cluster_name cluster_description

Note that some subject_ids may not be present in the prompt, and so should
also not be present in your response.

Provide both the (tab-separated) cluster-definitions table and the (tab-
separated) cluster-assignments table in your response.

\n"""

prompts = {

"default": """Cluster the responses in the table below highlighting the common
theme per cluster.

Group subjects that provide unclear, irrelevant, or no responses into a
separate "excluded" cluster.

Subjects can belong to multiple clusters. Your response should be in tab-
separated-values format,

with the following columns: subject_id, cluster_ids

Example output line:

subject_id cluster_ids

C-002 "C2,C3"

Start your response by defining each cluster in tab-separated-values format,
with columns:

cluster_id, cluster_name, cluster_description

Note that some subject_ids may not be present in the prompt, and so should
also not be present in your response.

Provide both the (tab-separated) cluster-definitions table and the (tab-
separated) cluster-assignments table in your response.

\n""",

0: TEMPLATE.format(

clusters="(1) Young Adults (22 to 33), (2) Middle-aged Adults (34 to 45), (3)
Older Adults (46 to 60), (4) Seniors (61 and above), and (5)
Unclear/irrelevant/no response"

),

1: TEMPLATE.format(

clusters="(1) Houston, Texas, (2) San Antonio, Texas, (3) Texas (Other), (4)
Florida, (5) Mid-West US, (6) US (Other) and (7) Unclear/Excluded/No response"

),

2: TEMPLATE.format(

clusters="(1) Not currently married, (2) Married currently, and (3)
Unclear/Excluded/No response"

),

14: TEMPLATE.format(

clusters="(1) Yes, (2) No, and (3) Unclear/irrelevant/no response"

),

16: TEMPLATE.format(

clusters="(1) Yes, (2) No, and (3) Unclear/irrelevant/no response"

),

# 17: Numeric: How many hours are you working on average (per week)?

17: TEMPLATE.format(

clusters="(1) Full-time, (2) Less than Full-time, (3) More than Full-time, and
(4) Unclear/Excluded/No response"

),

18: TEMPLATE.format(

clusters="(1) Increased hours, (2) Decreased hours, (3) No change, (4) Other,
and (5) Unclear/irrelevant/no response"

),

# 19: How does this compare to pre-covid-19 crisis?

19: TEMPLATE.format(

clusters="(1) Increased hours, (2) Decreased hours, (3) No change, (4) Other,
and (5) Unclear/irrelevant/no response"

),

20: TEMPLATE.format(

clusters="(1) Better, (2) Worse, (3) No-change, (4) Other and (5)
Unclear/irrelevant/no response"

),

21: TEMPLATE.format(

clusters="(1) Changed, (2) No change, (3) Fluctuating/uncertain change, and
(4) Unclear/irrelevant/no response"

),

22: TEMPLATE.format(

clusters="(1) Yes, (2) No, and (3) Unclear/irrelevant/no response"

),

23: TEMPLATE.format(

clusters="(1) Prepared, (2) Unprepared, and (3) Unclear/irrelevant/no
response"

),

24: TEMPLATE.format(

clusters="(1) Prepared, (2) Unprepared, and (3) Unclear/irrelevant/no
response"

),

25: TEMPLATE.format(

clusters="(1) Positively (e.g. excitement), (2) Negatively, (3) Mix of
Positively and Negatively, (4) Neutral, and (5) Unclear/irrelevant/no
response"

),

26: TEMPLATE.format(

clusters="(1) Yes, (2) No, (3) Mixed, (4) Fluctuating over time and (5)
Unclear/irrelevant/no response"

),

27: TEMPLATE.format(

clusters="(1) Yes, (2) No, and (3) Unclear/irrelevant/no response"

),

28: TEMPLATE.format(

clusters="(1) No/Mild (e.g. 1, 2 or 3 out of 10), (2) Moderate (e.g. 4, 5 or 6
out of 10), (3) Severe (e.g. 7, 8, 9 or 10 out of 10), and (4)
Unclear/irrelevant/no response"

),

29: TEMPLATE.format(

clusters="(1) No/Mild (e.g. 1, 2 or 3 out of 10), (2) Moderate (e.g. 4, 5 or 6
out of 10), (3) Severe (e.g. 7, 8, 9 or 10 out of 10), and (4)
Unclear/irrelevant/no response"

),

30: TEMPLATE.format(

clusters="(1) Positively (e.g. excitement), (2) Negatively, (3) Neutral/Mixed
and (4) Unclear/irrelevant/no response"

),

31: TEMPLATE.format(

clusters="(1) Yes, (2) No, (3) Mixed, and (4) Unclear/irrelevant/no response"

),

32: TEMPLATE.format(

clusters="(1) Positive, (2) Negative, (3) Neutral/Mixed and (4)
Unclear/irrelevant/no response"

),

33: TEMPLATE.format(

clusters="(1) Positive, (2) Negative, (3) Neutral/Mixed and (4)
Unclear/irrelevant/no response"

),

34: TEMPLATE.format(

clusters="(1) Yes, (2) No, and (3) Unclear/irrelevant/no response"

),

35: TEMPLATE.format(

clusters="(1) Yes, (2) No, and (3) Unclear/irrelevant/no response"

),

36: TEMPLATE.format(

clusters="(1) Yes, (2) No, and (3) Unclear/irrelevant/no response"

),

37: TEMPLATE.format(

clusters="(1) Yes will get professional help, (1) Yes but not professional
help, (3) Mixed, (4) Will not seek/get help and (5) Unclear/irrelevant/no
response"

),

38: TEMPLATE.format(

clusters="(1) Yes, (2) No, and (3) Unclear/irrelevant/no response"

),

39: TEMPLATE.format(

clusters="(1) Adhering Closely, (2) Not adhering closely OR Adhering
conditionally, and (3) Unclear/irrelevant/no response"

),

40: TEMPLATE.format(

clusters="(1) Yes, (2) No, (3) Mixed/Conditionally, and (3)
Unclear/irrelevant/no response"

),

}

##  Appendix C LLM prompt for Re-Clustering using previously defined clusters

⬇

Cluster the responses in the table below highlighting the common theme per
cluster.

Group subjects that provide unclear, irrelevant, or no responses into a
separate "excluded" cluster.

Subjects can belong to multiple clusters. Your response should be in tab-
separated-values format,

with the following columns: subject_id, cluster_ids

Example output line:

subject_id cluster_ids

C-002 "C2,C3"

Note that some subject_ids may not be present in the prompt, and so should
also not be present in your response.

Provide both the (tab-separated) cluster-definitions table and the (tab-
separated) cluster-assignments table in your response.

subject_id Are you a caretaker otherwise? (if not own kids, eg elderly
parents, adopted family member, etc)

C001 No

C002 No

C003 No

C004 No

C005 No

...

C086 Yes, looks after his mother-in-law’s finances

C087 No

C090 Yes; Partial caretaker for parents

C099 No

C100 No

C101 No

C102 No

Use the following cluster definitions (Do not repeat this in output):

cluster_id cluster_name cluster_description

C1 Caretakers of Family Members Subjects who responded that they take care of
relatives (elderly parents, children, siblings or others).

C2 Caretakers of Animals Subjects who take care of animals.

C3 Partial Caretakers Subjects who participate in caretaking but not as
primary caretakers.

C4 Financially Supportive Subjects who provide financial support instead of
physical caretaking.

C5 No Caretaking Responsibilities Subjects who stated that they do not take
care of anyone.

C6 Excluded Responses that are unclear, irrelevant, or did not provide a
response to the question.

##  Appendix D Concordance analysis

Figure 7:  Differences in subject and question concordance ratio accounts for
most inter-evaluator variability : (A) Average concordance score of subjects
compared across evaluator pairs. Average subject concordance compared between
evaluator pairs (E1 v E2, E1 v GPT, and E2 v GPT) using 2Way ANOVA found a
that the subject accounts for 57.4% of the variability (p $<$ 0.0001) while
differences between evaluator pairs account for 22.5% of the variability (p
$<$ 0.0001). Post-hoc Tukey’s multiple comparison across evaluator pairs found
that the subject concordance of E2 v GPT was significantly different than both
E1 v GPT and E1 v E2. (B) Average concordance score of questions compared
across evaluator pairs. Average question concordance compared between
evaluator pairs using 2Way ANOVA found a that the question accounts for 58.1%
of the variability (p $<$ 0.0001) while differences between evaluator pairs
account for 9.8% of the variability (p$<$0.0001). Post-hoc Tukey’s multiple
comparison across evaluator pairs found that the subject concordance of E2 v
GPT was significantly different than both E1 v GPT and E1 v E2. (C)
Concordance score of questions were averaged across categories (mean +/- SEM)
and compared across evaluator pairs. The average category concordance compared
between evaluator pairs using 2Way ANOVA found that the category accounts only
for 10.8% of the variability (p = 0.032) while differences between evaluator
pairs account for 14.6 % of the variability (p = 0.001). Post-hoc Tukey’s
multiple comparison across evaluator pairs found that the subject concordance
of E2 v GPT was significantly different than both E1 v GPT and E1 v E2. * p
$<$ 0.5, ** p $<$ 0.01, *** p $<$ 0.001, **** p $<$ 0.0001.

##  Appendix E Concordance vs Cohen’s Kappa

Figure 8:  Cohen’s kappa coefficient correlates with the concordance ratio:
(A) The Cohen’s Kappa coefficient calculated for each question correlates
significantly with the evaluator concordance score using Spearman rank
correlation. (B) The distribution of Cohen’s Kappa coefficient for each
question between different evaluator pairs differs (Friedman test, p = 0.0015
with Dunn’s multiple comparison). * p $<$ 0.5, *** p $<$ 0.01, *** p $<$
0.001, **** p $<$0.0001.


# Cross-Section Continuity of Definitions of Angular Momentum

Po-Ning Chen<EMAIL_ADDRESS>Department of Mathematics
University of California
Riverside, CA, USA.  Daniel E. Paraizo<EMAIL_ADDRESS>Enrico Fermi Institute
and Department of Physics
The University of Chicago
5640 South Ellis Avenue, Chicago, IL 60637, USA.  Institute for Gravitation
and the Cosmos and Department of Physics, Pennsylvania State University,
University Park, PA, 16802  Robert M. Wald<EMAIL_ADDRESS>Enrico Fermi
Institute and Department of Physics
The University of Chicago
5640 South Ellis Avenue, Chicago, IL 60637, USA.  Mu-Tao Wang
<EMAIL_ADDRESS>Department of Mathematics
Columbia University
New York, NY, USA.  Ye-Kai Wang<EMAIL_ADDRESS>Department of Applied
Mathematics
National Yang Ming Chiao Tung University
Hsinchu, Taiwan.  National Center of Theoretical Science, Taipei City, Taiwan
Shing-Tung Yau<EMAIL_ADDRESS>Department of Mathematics
Tsinghua University
Beijing, China.

###### Abstract

We introduce a notion of “cross-section continuity” as a criterion for the
viability of definitions of angular momentum, $J$, at null infinity: If a
sequence of cross-sections, ${\mathcal{C}}_{n}$, of null infinity converges
uniformly to a cross-section ${\mathcal{C}}$, then the angular momentum,
$J_{n}$, on ${\mathcal{C}}_{n}$ should converge to the angular momentum, $J$,
on ${\mathcal{C}}$. The Dray-Streubel (DS) definition of angular momentum
automatically satisfies this criterion by virtue of the existence of a well
defined flux associated with this definition. However, we show that the one-
parameter modification of the DS definition proposed by Compere and Nichols
(CN)—which encompasses numerous other alternative definitions—does not satisfy
cross-section continuity. On the other hand, we prove that the Chen-Wang-Yau
(CWY) definition does satisfy the cross-section continuity criterion.

###### Contents

  1. 1 Introduction
  2. 2 The DS, CN, and CWY Definitions of Angular Momentum
  3. 3 Failure of Cross-Section Continuity of the CN Modification of the DS Angular Momentum
  4. 4 Satisfaction of Cross-Section Continuity of the CWY Angular Momentum
    1. 1 Relating the fluxes of $J^{\rm CWY}$ and $J^{\rm DS}$
    2. 2 Estimating $\delta(f)$

##  1 Introduction

General relativity is a complete classical theory of gravity. It does not
require definitions of auxiliary quantities such as mass and angular momentum
in order to make physical predictions. Nevertheless, such auxiliary quantities
can be extremely useful for proving general results on the behavior of systems
as well as for obtaining physical insight into the nature of various
phenomena. In particular, the notion of Bondi mass—together with its
positivity and the positivity of its flux—provides an extremely powerful tool
for characterizing and constraining possible behaviors in asymptotically flat
spacetimes.

One would expect a notion of angular momentum also to be very useful for
characterizing the behavior of systems in asymptotically flat spacetimes.
However, a notable difficulty in obtaining a useful notion of angular momentum
arises from the fact that in general relativity the asymptotic symmetry group
is the Bondi-Metzner-Sachs (BMS) group. The BMS group significantly enlarges
the Poincare group by including “angle dependent translations” known as
supertranslations, which are characterized by an arbitrary function on a
sphere. As will be elucidated further in the next paragraph, in a general
asymptotically flat spacetime, there is no unique way of selecting a preferred
Poincare subgroup of the BMS group. Consequently, there is no obvious way of
distinguishing between a “pure rotation” and a “rotation plus
supertranslation.” Correspondingly, there is no obvious way of distinguishing
between “angular momentum” and “angular momentum plus a supertranslation
charge.”

The nature of the difficulty in selecting a preferred Poincare subgroup of
asymptotic symmetries in a generic asymptotically flat spacetime is elucidated
by the following considerations. During a sufficiently long era where the
Bondi news vanishes, there exists a unique $4$-parameter family of cross-
sections of null infinity— which we refer to as “electric parity good cuts”—on
which the electric parity part of the shear tensor111The usual notion of “good
cuts” requires that both the electric and magnetic parity parts of the shear
vanish. We require only the vanishing of the electric parity part of the
shear. vanishes NP . In such eras, there is a unique Poincare subgroup of the
BMS group that maps these electric parity good cuts into themselves NP . Thus,
in eras where the Bondi news vanishes, one can naturally eliminate the
“supertranslation ambiguity” of the BMS group by restricting to this Poincare
subgroup. In particular, if the news vanishes sufficiently rapidly at early
retarded times ($u\to-\infty$), one can naturally pick out a preferred
Poincare subgroup at asymptotically early times. Similarly, if the news
vanishes sufficiently rapidly at late retarded times ($u\to+\infty$), one can
naturally pick out a preferred Poincare subgroup at asymptotically late times.
The problem is that, generically, these early and late time subgroups will be
different Poincare subgroups of the BMS group. Indeed, the memory effect is
characterized by having the “electric parity good cuts” at asymptotically
early and late times differ by a nontrivial supertranslation. The fact that a
nonvanishing memory effect generically is present in asymptotically flat
spacetimes provides a clear demonstration that, generically, it cannot be
useful to attempt to restrict consideration to a single Poincare subgroup of
asymptotic symmetries.

Given the above situation, there are two possible strategies that can be
employed to attempt to define a notion of angular momentum. The first is to
abandon the attempt to identify a unique notion of a “rotation” and a
corresponding “angular momentum” but rather work with the entire group of BMS
symmetries. One then defines notions of “charge” conjugate to all BMS
symmetries. Of course, any individual might choose to declare a particular BMS
symmetry to be a “rotation” and they might then refer to the corresponding BMS
charge as “angular momentum.” However, different individuals may make
different choices. In other words, this approach has the drawback that there
would be an inherent supertranslation ambiguity in what BMS charge should be
called “angular momentum.” Furthermore, as described above, at asymptotically
early and late retarded times, one does have a well defined notion of a “pure
rotation” (as opposed to a “rotation plus supertranslation”). However, in the
presence of memory, no fixed BMS charge can simultaneously correspond to this
natural notion of angular momentum in both of these asymptotic regions.

A second strategy is to use data on a given cross-section to effectively
determine what should be considered to be a “pure rotation” and “angular
momentum” at the particular retarded time represented by that cross-section.
In this way, the supertranslation ambiguity would be eliminated and—since the
definition does not require use of a fixed BMS symmetry—it can be chosen to
correspond to the well defined notions of “pure rotation” and “angular
momentum” in eras when the Bondi news vanishes. Thus, this strategy has the
potential to yield a notion of angular momentum in general relativity with
properties much more analogous to the notion of angular momentum in non-
gravitational physics. A potential drawback of this strategy is that there may
be circumstances where there is a distinguished BMS symmetry—such as when the
spacetime admits an axial Killing field—and there is no guarantee that this
approach will yield a quantity that can be interpreted as the charge
associated with this symmetry.

A successful implementation of the first strategy was given by Dray and
Streubel (DS) DS . As shown in WZ , the formulas for the DS charges conjugate
to the BMS symmetries can be given a strong motivation from Hamiltonian
considerations. Nevertheless, possibilities remain open for modifying the DS
formulas for the BMS charges. Recently, Compere and Nichols (CN) CN  have
proposed a one-parameter modification of the DS definition. This modification
encompasses other definitions that had been given previously LL -CFR . Thus,
there remain questions as to whether the DS definition is the only viable
definition within the context of the first strategy.

A successful implementation of the second strategy was given by Chen-Wang-Yau
(CWY) CWY . As shown in CWWY ; CKWWY , the limit of the Chen-Wang-Yau
quasilocal angular momentum at null infinity (evaluated in KWY ) is free of
supertranslation ambiguity. In addition, the CWY angular momentum transforms
according to classical laws with respect to “ordinary translations” (CWWY, ,
eq.(22)).

The main purpose of this paper is to subject the above definitions to the
criterion of “cross-section continuity”: If an arbitrarily small amplitude
“wiggling” of a cross-section in retarded time can produce finite changes in
the angular momentum, then that definition is not viable, since it would be
telling one at least as much about the choice of cross-section as any physical
properties of the spacetime. In section 2, we review the definitions of the
DS, CN, and CWY angular momentum and we then introduce the notion of cross-
section continuity. The DS definition can be seen to satisfy this condition by
virtue of the existence of a well-defined flux. However, in section 3, we show
that the CN definition does not satisfy this condition except for the choice
of parameter where it coincides with the DS definition. Thus, the CN
generalization is not viable. Finally, in section 4, we prove that the CWY
definition does satisfy the cross-section continuity requirement.

##  2 The DS, CN, and CWY Definitions of Angular Momentum

We will work in Bondi coordinates $(u,r,x^{A})$, where $x^{A}$ denotes angular
coordinates on the $2$-sphere. We will follow the conventions of Flanagan and
Nichols FN  and refer the reader to that reference for the definitions of the
shear tensor, $C_{AB}(u,x^{A})$, mass aspect, $m(u,x^{A})$, and angular
momentum aspect, $N_{A}(u,x^{A})$. The Bondi news tensor is defined by
$N_{AB}=\partial_{u}C_{AB}$. We denote the round metric on the sphere as
$q_{AB}$. We denote the corresponding derivative operator on the sphere as
$D_{A}$, and we write $D^{2}=D^{A}D_{A}$.

Our analysis could be performed for arbitrary BMS symmetries. However, to keep
both the discussion and the formulas as simple as possible, we will restrict
consideration here to the BMS symmetry

$\displaystyle X=Y^{A}\frac{\partial}{\partial x^{A}}$ (1)

where $Y^{A}$ is a fixed rotational Killing field on the spheres of constant
$u$. This symmetry corresponds to a “pure rotation” in our Bondi coordinates
$(u,r,x^{A})$. We will refer to the charge corresponding to this symmetry as
“angular momentum.” However, we will be interested in evaluating the angular
momentum on an arbitrary cross-section given by

$\displaystyle u=f(x^{A})$ (2)

where $f:S^{2}\to{\mathbb{R}}$ is smooth. Note that X does not act
tangentially to such a cross-section. Indeed, in the new Bondi coordinates

$\displaystyle u^{\prime}$ $\displaystyle=$ $\displaystyle u-f(x^{A})$
$\displaystyle x^{\prime A}$ $\displaystyle=$ $\displaystyle x^{A}$ (3)

for which the cross-section is given by $u^{\prime}=0$, we have

$\displaystyle X=Y^{A}\frac{\partial}{\partial x^{\prime
A}}-Y^{A}D_{A}f\frac{\partial}{\partial u^{\prime}}\,.$ (4)

Thus, in the new Bondi coordinates, $X$ corresponds to the rotation $Y^{A}$
plus a supertranslation by the amount $-Y^{A}D_{A}f$. This illustrates the
supertranslation ambiguity in defining “rotations” that was described in the
Introduction.

As discussed in the Introduction, the DS definition does not attempt to
resolve the supertranslation ambiguity and simply defines an angular momentum
quantity $J^{\rm DS}$ conjugate to the BMS symmetry $X$. On the cross-section
$u=0$, the formula for the DS angular momentum is222A formula for the DS
charges in Bondi coordinates is given in eq.(3.5) of FN . A restriction to
vanishing news was made in the section of FN  containing that equation but, as
shown in GPS , the charge formula is valid without this restriction.

$\displaystyle J^{\rm DS}|_{u=0}=\frac{1}{8\pi}\int
Y^{A}\bigg{(}N_{A}-\frac{1}{4}C_{AB}D_{D}C^{DB}\bigg{)}$ (5)

where all quantities in the integrand are evaluated at $u=0$ and the integral
is taken over the sphere with the natural volume element associated with
$q_{AB}$. (In the following, unless otherwise noted, all integrals are taken
over a sphere with this volume element.) We can evaluate $J^{\rm DS}$ on the
cross-section $u=f$ by transforming to the new Bondi coordinates eq. (3) and
taking into account the fact that $X$ has an additional supertranslation part
in these coordinates (see eq. (4)). On the cross-section $u=f$, the formula
for DS angular momentum is

$\displaystyle J^{\rm DS}|_{u=f}=\frac{1}{8\pi}\int
Y^{A}\bigg{(}N^{\prime}_{A}-\frac{1}{4}Y^{A}C^{\prime}_{AB}D_{D}C^{\prime
DB}\bigg{)}-\frac{1}{4\pi}\int m^{\prime}Y^{A}D_{A}f$ (6)

where the last term is the supertranslation contribution. The quantities
$C^{\prime}_{AB}$, $m^{\prime}$, and $N^{\prime}_{A}$ are the shear tensor,
mass aspect, and angular momentum aspect in the new Bondi coordinates eq. (3)
evaluated at $u^{\prime}=0$ (i.e., $u=f$). Explicitly, we have (see section
C.5 of CJK )

$\displaystyle C^{\prime}_{AB}$ $\displaystyle=$ $\displaystyle
C_{AB}\big{|}_{u=f}-2D_{A}D_{B}f+q_{AB}D^{2}f$ (7) $\displaystyle m^{\prime}$
$\displaystyle=$ $\displaystyle
m|_{u=f}+\frac{1}{4}N^{AB}\big{|}_{u=f}D_{A}D_{B}f+\frac{1}{2}[D_{B}N^{AB}]\big{|}_{u=f}D_{A}f+\frac{1}{4}[\partial_{u}N^{AB}]\big{|}_{u=f}D_{A}fD_{B}f.$
(8)

There is a similar (although considerably more complicated) formula for
$N^{\prime}_{A}$ but we will not need this formula in our analysis below. Note
that it follows immediately from eq. (7) that the news tensor transforms
simply as

$\displaystyle N^{\prime}_{AB}=N_{AB}|_{u=f}.$ (9)

The CN definition of angular momentum follows the same strategy as DS of not
attempting to resolve the supertranslation ambiguity but it makes a
modification of the DS formula by the addition of a term with an arbitrary
free parameter $\alpha$. For the BMS symmetry $X$ given by eq.(1), the CN
angular momentum at $u=0$ is given by CN

$\displaystyle J^{\rm CN}|_{u=0}=J^{\rm DS}|_{u=0}-\frac{\alpha-1}{32\pi}\int
Y^{A}C_{AB}D_{C}C^{BC}\,.$ (10)

This coincides with the DS angular momentum when $\alpha=1$. As Compere and
Nichols have noted CN , the choice $\alpha=3$ corresponds to the definitions
used by Landau and Lifshitz LL , Bonga and Poisson BP , and Damour Dam ,
whereas the choice $\alpha=0$ corresponds to the definitions used by
Pasterski, Strominger, and Zhiboedov PSZ  and Compere, Fiorucci, and Ruzziconi
CFR . On the cross-section $u=f$, the CN angular momentum is given by

$\displaystyle J^{\rm CN}|_{u=f}=J^{\rm DS}|_{u=f}-\frac{\alpha-1}{32\pi}\int
Y^{A}C^{\prime}_{AB}D_{C}C^{\prime BC}\,.$ (11)

As described in the Introduction, the CWY definition of angular momentum
resolves the supertranslation ambiguity by, in effect, determining what
corresponds to a “pure rotation” on any cross-section from the data on that
cross-section. To define CWY angular momentum on any cross-section, one does
not need to select a BMS symmetry $X$ but merely a Killing field $Y^{A}$ on
the sphere. On the cross-section $u=0$, the CWY angular momentum associated
with $Y^{A}$ is related to the DS angular momentum by CWWY ; CKWWY

$\displaystyle J^{\rm CWY}|_{u=0}=J^{\rm DS}|_{u=0}+\frac{1}{8\pi}\int
mY^{A}D_{A}c$ (12)

where $J^{\rm DS}$ is the DS angular momentum associated with $X$, eq.(1), and
$c$ is the unique solution to

$\displaystyle\frac{1}{2}D^{2}(D^{2}+2)c=D^{A}D^{B}C_{AB}$ (13)

such that $c$ has no $\ell=0,1$ parts. Thus, the CWY angular momentum
corresponds to the DS charge conjugate to the BMS symmetry

$\displaystyle\tilde{X}=Y^{A}\frac{\partial}{\partial
x^{A}}+\frac{1}{2}Y^{A}D_{A}c\frac{\partial}{\partial u}$ (14)

and, from this perspective, it can be viewed as selecting $\tilde{X}$ as being
the BMS symmetry representing a “pure rotation” at the time $u=0$. Comparing
with eq. (4), we see that $\tilde{X}$ is tangent to the cross-section $u=c/2$.
Now, in a sufficiently long non-radiative era—where the Bondi News vanishes
and thus $C_{AB}$ is independent of $u$—it follows from eq. (7) that the
electric parity part of the shear tensor vanishes on the cross-section
$u=c/2$. Thus, in a non-radiative era, $\tilde{X}$ is tangent to an “electric
parity good cut” of null infinity and thus represents a natural choice of
“pure rotation.” In particular, the CWY definition corresponds to the natural
choices of “pure rotation” at both asymptotically early and late retarded
times—even though these early and late time choices correspond to different
BMS symmetries. The CWY angular momentum on an arbitrary cross section
interpolates between these choices in a manner that depends on the local
conditions at that cross-section. However, if the spacetime has an exact axial
Killing field, we have shown that $\tilde{X}$ need not correspond to this
Killing field when the Bondi news is nonvanishing, i.e., the CWY angular
momentum need not coincide with the DS (= Komar) angular momentum in the case
where $X$ corresponds to an exact Killing field in the spacetime. In this
case,

$\displaystyle J^{\rm DS}|_{u=f}=J^{\rm DS}|_{u=0}\text{ for any cross-section
$u=f$}.$ (15)

On the other hand,

$\displaystyle J^{\rm CWY}|_{u=f}=J^{\rm CWY}|_{u=0}\text{ only if $f$
satisfies $Y^{A}D_{A}f=0$. }$ (16)

On the cross-section $u=f$, a formula for the CWY angular momentum can be
obtained by transforming to the new Bondi coordinates eq. (3). We obtain

$\displaystyle J^{\rm CWY}|_{u=f}=J^{\rm DS}|_{u=f}+\frac{1}{8\pi}\int
m^{\prime}Y^{A}D_{A}c^{\prime}+\frac{1}{4\pi}\int m^{\prime}Y^{A}D_{A}f$ (17)

where $c^{\prime}$ is defined by eq. (13) with $C_{AB}$ replaced by
$C^{\prime}_{AB}$.

We now introduce our cross-section continuity condition. Let
$J:{\mathscr{C}}\to{\mathbb{R}}$ be any map from the space, $\mathscr{C}$, of
smooth cross-sections of null infinity into $\mathbb{R}$. Then $J$ will be
said to satisfy the cross-section continuity condition at $u=0$ if for any
sequence $\\{f_{n}\\}$ of smooth functions on the sphere such that $f_{n}\to
0$ uniformly as $n\to\infty$ we have $J|_{u=f_{n}}\to J|_{u=0}$. If this
condition is not satisfied, then the value of $J$ would depend on the fine
details of the choice of cross-section and could not plausibly provide useful
physical information about the properties of the spacetime. Our goal for the
remainder of this paper is to determine whether $J^{\rm DS}$, $J^{\rm CN}$,
and $J^{\rm CWY}$ satisfy the cross-section continuity condition.

Since the formulas (6), (11), and (17) for $J^{\rm DS}$, $J^{\rm CN}$, and
$J^{\rm CWY}$ on the cross-section $u=f$ depend nonlinearly on derivatives of
$f$, it is not immediately obvious by inspection of these formulas whether
cross-section continuity holds for any of these quantities. However, it is
known that the DS formula has an associated flux WZ , GPS . For the BMS
symmetry $X$ given by eq.(1), the flux in our original Bondi coordinates is
given by FN

$\displaystyle{\mathcal{F}}=-\frac{1}{32\pi}N^{AB}\pounds_{Y}C_{AB}.$ (18)

For any two cross-sections
${\mathcal{C}}_{1},{\mathcal{C}}_{2}\in{\mathscr{C}}$, the difference between
the DS angular momenta on these cross-sections is given by

$\displaystyle J^{\rm DS}|_{{\mathcal{C}}_{2}}-J^{\rm
DS}|_{{\mathcal{C}}_{1}}=\int_{\mathcal{R}}{\mathcal{F}}$ (19)

where $\mathcal{R}$ denotes the (compact) region of null infinity bounded by
${\mathcal{C}}_{1}$ and ${\mathcal{C}}_{2}$. It follows immediately from this
flux formula—taking ${\mathcal{C}}_{2}$ to be the cross-section $u=f$ and
${\mathcal{C}}_{1}$ to be the cross-section $u=0$—that cross-section
continuity is satisfied by the DS definition of angular momentum. In the next
section, we shall show that cross-section continuity does not hold for $J^{\rm
CN}$ except in the case $\alpha=1$, where the CN definition coincides with the
DS definition. In section 4, we will show that cross-section continuity does
hold for the CWY definition of angular momentum.

##  3 Failure of Cross-Section Continuity of the CN Modification of the DS
Angular Momentum

Since $J^{\rm DS}$ satisfies the cross-section continuity condition, it is
clear that $J^{\rm CN}$ satisfies this condition if and only if the quantity

$\displaystyle K\equiv J^{\rm CN}-J^{\rm DS}$ (20)

satisfies this condition. We have

$\displaystyle K|_{u=f}-K|_{u=0}=-\frac{\alpha-1}{32\pi}\int
Y^{A}\left[C^{\prime}_{AB}D_{C}C^{\prime BC}-C_{AB}D_{C}C^{BC}\right].$ (21)

Let $(\theta,\phi)$ denote the usual spherical coordinates on the sphere. We
consider the Killing field $Y^{A}=(\partial/\partial\phi)^{A}$ and investigate
whether the right side converges to zero for the choice of sequence

$f_{n}(\theta,\phi)=\frac{1}{n}F(\theta)\sin(n\phi)$ (22)

where $F$ is a smooth function of $\theta$ that vanishes in a neighborhood of
$\theta=0$ and $\theta=\pi$. Note that any expression containing at least $p$
factors of $f_{n}$ and a total of at most $q$ angular derivatives, $D_{A}$, is
$O(1/n^{p-q})$ as $n\to\infty$, so any term with more factors of $f_{n}$ than
angular derivatives can be neglected in this limit. Taylor expanding
$C_{AB}\big{|}_{u=f_{n}}$ in eq. (7), we obtain

$C^{\prime}_{AB}=C_{AB}\big{|}_{u=0}+f_{n}N_{AB}\big{|}_{u=0}+\frac{1}{2}f_{n}^{2}\partial_{u}N_{AB}\big{|}_{u=0}+O(f_{n}^{3})-2D_{A}D_{B}f_{n}+q_{AB}D^{2}f_{n}$
(23)

where the $O(f_{n}^{3})$ term arises from the Taylor expansion and does not
contain angular derivatives of $f_{n}$. This $O(f_{n}^{3})$ term appears on
the right side of eq. (21) only in terms that contain more factors of $f_{n}$
than angular derivatives, so for the sequence (22), the $O(f_{n}^{3})$ term
cannot contribute to the right side of eq. (21) as $n\to\infty$. The term
$\frac{1}{2}f_{n}^{2}\partial_{u}N_{AB}|_{u=0}$ potentially could contribute
as $n\to\infty$ via terms that are cubic in $f_{n}$ with a total of three
$\phi$-derivatives acting on $f_{n}$. An example of such a term is

$\displaystyle\frac{1}{2}Y^{A}f_{n}^{2}\partial_{u}N_{AB}(-$ $\displaystyle 2$
$\displaystyle D_{C}D^{B}D^{C}f_{n})\approx-
f_{n}^{2}\partial_{u}N_{\phi\phi}(q^{\phi\phi})^{2}D_{\phi}D_{\phi}D_{\phi}f_{n}$
(24) $\displaystyle\approx$ $\displaystyle-
f_{n}^{2}\partial_{u}N_{\phi\phi}\frac{1}{\sin^{4}\theta}\frac{\partial^{3}f_{n}}{\partial\phi^{3}}\approx
F^{3}(\theta)\partial_{u}N_{\phi\phi}\frac{1}{\sin^{4}\theta}\sin^{2}(n\phi)\cos(n\phi)$

where $\approx$ denotes equality modulo terms that vanish for the sequence
(22) as $n\to\infty$. Although the right side of eq.(24) remains bounded as
$n\to\infty$, its integral over a sphere vanishes as $n\to\infty$. Similarly,
all other terms involving $\frac{1}{2}f_{n}^{2}\partial_{u}N_{AB}|_{u=0}$ do
not contribute as $n\to\infty$. The remaining terms on the right side of eq.
(21) are either linear in $f_{n}$ or quadratic in $f_{n}$. For the terms that
are linear in $f_{n}$, all derivatives acting on $f_{n}$ can be removed by
integration-by-parts, in which case it is clear that these terms cannot
contribute as $n\to\infty$. The right side of eq.(21) is thus reduced to the
quadratic terms

$-\frac{\alpha-1}{32\pi}\int
Y^{A}\bigg{[}f_{n}N_{AB}D_{C}(f_{n}N^{BC})+\frac{1}{2}C_{AB}D_{C}(f^{2}_{n}\partial_{u}N^{BC})-2f_{n}N_{AB}D_{C}D^{B}D^{C}f_{n}\\\
+f_{n}N_{AB}D^{B}D^{2}f_{n}+\frac{1}{2}f_{n}^{2}\partial_{u}N_{AB}D_{C}C^{BC}-2D_{A}D_{B}f_{n}D_{C}(f_{n}N^{BC})+D^{2}f_{n}D^{B}(f_{n}N_{AB})\\\
+4D_{A}D_{B}f_{n}D_{C}D^{B}D^{C}f_{n}-2D_{A}D_{B}f_{n}D^{B}D^{2}f_{n}-2D^{2}f_{n}D_{B}D_{A}D^{B}f_{n}+D^{2}f_{n}D_{A}D^{2}f_{n}\bigg{]}.$
(25)

The terms appearing in (25) can be broken into 3 types: (i) terms quadratic in
the news; (ii) terms linear in the news; (iii) terms that are independent of
the news. The terms quadratic in the news contain at most one derivative of
$f_{n}$ and cannot contribute to the right side of eq. (21) as $n\to\infty$.
The terms that are independent of the news can be shown to give vanishing
contribution by a calculation similar to the calculation that shows that the
CN angular momentum vanishes for an arbitrary cross-section in Minkowski
spacetime (see section III of EN ). Thus, we need only consider the
contribution to the right side of eq. (21) arising from terms that are linear
in the news and quadratic in $f_{n}$. After some cancelations arising from
integration by parts, we obtain

$K|_{u=f_{n}}-K|_{u=0}\approx\frac{\alpha-1}{16\pi}\int
Y^{A}\left[f_{n}N_{AB}D_{C}D^{B}D^{C}f_{n}+D_{A}D_{B}f_{n}D_{C}(f_{n}N^{BC})\right]\,.$
(26)

In the first term of (26), we may write $D_{C}D^{B}\approx D^{B}D_{C}$ since
the difference yields a curvature term that will not contribute as
$n\to\infty$. Integrating this term by parts with respect to $D^{B}$ after
this interchange, we obtain

$K|_{u=f_{n}}-K|_{u=0}\approx\frac{\alpha-1}{16\pi}\int
Y^{A}\left[-q_{AB}D^{2}f_{n}+D_{A}D_{B}f_{n}\right]D_{C}(f_{n}N^{BC}).$ (27)

Evaluating the right side for our choices of $Y^{A}$ and of $f_{n}$ and
discarding terms that do not contribute as $n\to\infty$, we obtain

$K|_{u=f_{n}}-K|_{u=0}\approx\frac{\alpha-1}{16\pi}\int
N^{\theta\phi}\cos^{2}(n\phi)\left[F\partial_{\theta}F-F^{2}\cot\theta\right]\,.$
(28)

Thus, we find

$\lim_{n\to\infty}K|_{u=f_{n}}-K|_{u=0}=-\frac{\alpha-1}{64\pi}\int
F^{2}(\theta)\left(\frac{\partial N^{\theta\phi}}{\partial\theta}+3\cot\theta
N^{\theta\phi}\right)\,.$ (29)

Since $F(\theta)$ is an arbitrary smooth function that vanishes near the
poles, it is clear that the integral on right side of eq. (29) is nonvanishing
in general, so the CN definition of angular momentum does not satisfy the
cross-section continuity condition except in the case $\alpha=1$, when it
coincides with the DS definition.

##  4 Satisfaction of Cross-Section Continuity of the CWY Angular Momentum

###  1 Relating the fluxes of $J^{\rm CWY}$ and $J^{\rm DS}$

We prove the cross-section continuity of the CWY angular momentum in the
following sense: The CWY angular momentum of a smooth cross section $u=f$
approaches the CWY angular momentum of the cross section $u=0$, provided $f$
approaches $0$ in $C^{0}$. Here $f$ is any smooth function on $S^{2}$. Such a
continuity statement precipitates the extension of the definition of CWY
angular momentum to all $C^{0}$ cross sections. The proof will rely on the
cross-section continuity of the DS angular momentum.

The strategy of the continuity proof consists of the following two steps.

In step 1), assuming $f(x^{A})>0$, we relate the fluxes of CWY angular
momentum and DS angular momentum in the region between $u=0$ and $u=f$. In
particular, we identify $\delta(f)$ in

$J^{\rm CWY}|_{u=f}-J^{\rm CWY}|_{u=0}={J}^{\rm DS}|_{u=f}-{J}^{\rm
DS}|_{u=0}+\delta(f)$ (30)

as an integral on $S^{2}$.

In step 2), we apply elliptic estimates to $\delta(f)$. In particular, we show
that

$\delta(f)\leq C||f||_{C^{0}}$

for a constant $C$. By the continuity of the DS angular momentum and the above
two steps, the proof of the cross-section continuity of the CWY angular
momentum will be completed.

First, combining eq.(12) and eq.(17) yields

$\begin{split}&J^{\rm CWY}|_{u=f}-J^{\rm CWY}|_{u=0}\\\ &={J}^{\rm
DS}|_{u=f}-{J}^{\rm DS}|_{u=0}+\frac{1}{4\pi}\int
m^{\prime}Y^{A}D_{A}f+\frac{1}{8\pi}\int
m^{\prime}Y^{A}D_{A}c^{\prime}-\frac{1}{8\pi}\int mY^{A}D_{A}c.\end{split}$
(31)

Therefore, we identify $\delta(f)$ as

$\delta(f)=\frac{1}{8\pi}\int
m^{\prime}Y^{A}D_{A}(c^{\prime}+2f)-\frac{1}{8\pi}\int mY^{A}D_{A}c.$ (32)

We proceed to relate $c^{\prime}$ and $c$. Introducing the unique function $s$
on $S^{2}$ of $\ell\geq 2$ that satisfies

$\frac{1}{2}D^{2}(D^{2}+2)s=D^{A}D^{B}(C_{AB}|_{u=f}-C_{AB}|_{u=0}),$ (33)

eq.(7) implies $c^{\prime}=s+c-2f$. Therefore, $\delta(f)$ can be written as

$\delta(f)=\frac{1}{8\pi}\int m^{\prime}Y^{A}D_{A}(s+c)-\frac{1}{8\pi}\int
mY^{A}D_{A}c.$ (34)

Finally, we relate $m^{\prime}$ to $s$. Recalling the modified mass aspect

$\widehat{m}(u,x^{A})=(m-\frac{1}{4}D^{A}D^{B}C_{AB})(u,x^{A})$ (35)

as in CKWWY , eq.(8) is then equivalent to

$m^{\prime}=\widehat{m}|_{u=f}+\frac{1}{8}D^{2}(D^{2}+2)(s+c).$ (36)

###  2 Estimating $\delta(f)$

In this subsection, we show that $\delta(f)$ is bounded by $\|f\|_{C^{0}}$. We
first rewrite eq.(34) as

$\displaystyle 8\pi\delta(f)$ $\displaystyle=\int
m^{\prime}Y^{A}D_{A}s+\int(m^{\prime}-m)Y^{A}D_{A}c$
$\displaystyle=\delta_{1}+\delta_{2}$

The following identity for a smooth function $g$ on $S^{2}$ is proved in
(CKWWY, , Lemma 2.1):

$\int(D^{2}(D^{2}+2)g)Y^{A}D_{A}g=0.$ (37)

By eq.(37), eq.(36), and integration by parts,

$\delta_{1}=\int(\widehat{m}|_{u=f}+\frac{1}{8}D^{2}(D^{2}+2)c)Y^{A}D_{A}s$

and

$\begin{split}\delta_{2}&=\int(\widehat{m}|_{u=f}-m+\frac{1}{8}D^{2}(D^{2}+2)c)Y^{A}D_{A}c+\frac{1}{4}\int(C_{AB}|_{u=f}-C_{AB}|_{u=0})D^{A}D^{B}(Y^{E}D_{E}c)\\\
&=\int\left[\int_{0}^{f}\partial_{u}\widehat{m}\,du\right]Y^{A}D_{A}c+\frac{1}{4}\int\left[\int_{0}^{f}\partial_{u}C_{AB}\,du\right]D^{A}D^{B}(Y^{E}D_{E}c),\end{split}$

where we used $m-\frac{1}{8}D^{2}(D^{2}+2)c=\widehat{m}|_{u=0}$. Recalling
$\partial_{u}\widehat{m}=-\frac{1}{8}N_{AB}N^{AB}$ from (CKWWY, , (3.3)), it
is clear that $|\delta_{2}|$ is bounded by a multiple of $\|f\|_{C^{0}}$.

On the other hand, we claim that

$|\delta_{1}|\leq C_{1}\|f\|_{C^{0}}$

by the following Lemma.

######  Lemma 4.1.

If $s$ satisfies $\frac{1}{2}D^{2}(D^{2}+2)s=D^{A}D^{B}M_{AB}$, then

$\displaystyle\|Ds\|_{L^{2}}\leq\sqrt{\frac{16\pi}{3}}\max_{S^{2}}|M_{AB}|.$

###### Proof.

Multiplying the equation by $s$ and integrating over $S^{2}$, we get

$\int(D^{2}s)^{2}-2|Ds|^{2}=\int 2M_{AB}D^{A}D^{B}s\leq\int
2|M_{AB}|^{2}+\int\frac{1}{2}|D_{A}D_{B}s|^{2}.$

Recalling the identity on $S^{2}$,

$\displaystyle\int(D^{2}s)^{2}=\int|D_{A}D_{B}s|^{2}+|Ds|^{2},$ (38)

we get

$\int(D^{2}s)^{2}-3|Ds|^{2}\leq 16\pi(\max_{S^{2}}|M_{AB}|^{2}).$

Let $s=\sum_{\ell=2}^{\infty}s_{\ell}$ be the spherical harmonic decomposition
of $s$. We have
$\int(D^{2}s)^{2}=\sum_{\ell=2}^{\infty}\ell^{2}(\ell+1)^{2}s_{\ell}^{2}$,
$\int|Ds|^{2}=\int-sD^{2}s=\sum_{\ell=2}^{\infty}\ell(\ell+1)s_{\ell}^{2}$ and
hence $\int(D^{2}s)^{2}\geq 6\int|Ds|^{2}$. As a result, we have
$\int|Ds|^{2}\leq\frac{16\pi}{3}\max_{S^{2}}|M|^{2}$. ∎

## Acknowledgements

This material is based upon work supported by the National Science Foundation
under Grant Number DMS-2104212 (Mu-Tao Wang) and Grant PHY-2105878 (D.E.
Paraizo and R.M. Wald), by the Simons Foundation under Grant Number 584785
(Po-Ning Chen), and by Taiwan NSTC grant 109-2628-M-006-001-MY3 (Y.-K. Wang).

## References

  * (1) E.T. Newman and R. Penrose, J. Math. Phys. 7, 863 (1966).
  * (2) T. Dray and M. Streubel, Class. Quant. Grav. 1, 15 (1984).
  * (3) R.M. Wald and A. Zoupas, Phys. Rev. D61, 084027 (2000) (arXiv:gr-qc/9911095).
  * (4) G. Compere and D. Nichols, arXiv:2103.17103.
  * (5) L. D. Landau and E. M. Lifshitz, The Classical Theory of Fields (Butterworth-Heinemann, 1975).
  * (6) B. Bonga and E. Poisson, Phys. Rev. D99, 064024 (2019) (arXiv:1808.01288).
  * (7) T. Damour, Phys. Rev. D102, 124008 (2020) (arXiv:2010.01641).
  * (8) S. Pasterski, A. Strominger, and A. Zhiboedov, JHEP 12, 053 (2016) (arXiv:1502.06120).
  * (9) G. Compere, A. Fiorucci, and R. Ruzziconi, JHEP 10, 205 (2020) (arXiv:2004.10769) and JHEP 11, 200 (2018), [Erratum: JHEP 04, 172 (2020)] (arXiv:1810.00377).
  * (10) P.-N. Chen, M.-T. Wang and S.-T. Yau, Comm. Math. Phys. 338 (2015), no.1, 31–80 and Adv. Theor. Math. Phys. 20 (2016), no. 4, 671–682.
  * (11) P.-N. Chen, M.-T. Wang, Y.-K. Wang, and S.-T. Yau, Adv. Theor. Math. Phys. 25 (2021), no. 3, 777–790.
  * (12) P.-N. Chen, J. Keller, M.-T. Wang, Y.-K. Wang, and S.-T. Yau, Comm. Math. Phys. 386 (2021), no. 1, 551–588.
  * (13) J. Keller, Y.-K. Wang, and S.-T. Yau, Adv. Theor. Math. Phys. 24 (2020), no. 6, 1423–1473.
  * (14) E.E. Flanagan and D.A. Nichols, Phys. Rev. D95, 044002 (2017) (arXiv:1510.03386).
  * (15) A.M. Grant, K. Prabhu, and I. Shehzad, arXiv:2105.05919.
  * (16) P.T. Chrusciel, J. Jezierski, and J. Kijowski, Hamiltonian Field Theory in the Radiating Regime, Lecture Notes in Physics. Monographs, 70 (Springer-Verlag, Berlin, 2002).
  * (17) A. Elhashash and D. Nichols, arXiv:2101.12228.