# FaaF: Facts as a Function for the evaluation of RAG systems

Vasileios Katranidis Gabor Barany
IMMO Capital
{vasileios.katranidis<EMAIL_ADDRESS>
###### Abstract

Factual recall from a reference source is crucial for evaluating the
performance of Retrieval Augmented Generation (RAG) systems, as it directly
probes into the quality of both retrieval and generation. However, it still
remains a challenge to perform this evaluation reliably and efficiently.
Recent work has focused on fact verification via prompting language model (LM)
evaluators, however we demonstrate that these methods are unreliable in the
presence of incomplete or inaccurate information. We introduce Facts as a
Function (FaaF), a new approach to fact verification that utilizes the
function calling abilities of LMs and a framework for RAG factual recall
evaluation. FaaF substantially improves the ability of LMs to identify
unsupported facts in text with incomplete information whilst improving
efficiency and lowering cost by several times, compared to prompt-based
approaches.

##  1 Introduction

The adoption and transformative impact of large language models (LMs) across
industries are significantly propelled by their application in knowledge-
intensive tasks.

Figure 1: An overview of FaaF, a constructor dynamically creates a function
object from a set of fact statements. Function calling allows LMeval to verify
all facts within a single call when provided with an input text. FaaF reduces
the error rate in identifying unsupported facts by up to 40 percentage points
compared to prompting whilst reducing the number of LMeval calls and output
tokens by more than 5 times.

Retrieval Augmented Generation (RAG) enables seamlessly the integration of
out-of-training knowledge to the LM (Lee et al., 2019). This is achieved by
dynamically fetching and incorporating relevant data from a trusted reference
source prior LM text generation, resulting in more informed, accurate, and
contextually rich outputs. In such applications, factual recall —the extent to
which all the information in the reference source that is required to
sufficiently answer the question is retrieved and correctly conveyed in the
response of the RAG— is a key performance metric because it closely reflects
the central purpose of these systems and captures the the performance of
retrieval and generation simultaneously.

Although there is a considerable body of work (Chen et al., 2022; Zhang et
al., 2023; Gao et al., 2023; Lee et al., 2023; Min et al., 2023; Azaria and
Mitchell, 2023; Yuan et al., 2021; Fu et al., 2023; Wang et al., 2023;
Kadavath et al., 2022; Manakul et al., 2023) that is focusing at factual
precision —the truthfulness of each statement in a generated text— factual
recall at least as important for RAG systems, where a response can be
factually precise but given the wrong context may be irrelevant to the posed
question.

Practical factual recall evaluation in RAG systems is still an open problem
because of the following reasons. First, the scenario of automatic
verification of truthful, independent statements, given a highly incomplete or
inaccurate generated text (simulating recall in a low quality RAG) has not
been as explored as precision on LM generated text. Secondly, considering that
there can be several facts which need to be verified in a single generated
text (E.g. a RAG response to a question), current approaches relying on
verification of each fact independently can be prohibitively costly in time
and resources. RAG systems include many moving parts (knowledge base,
retrieval, prompt formulation, LM) and require substantial tuning (Es et al.,
2023) therefore efficiency is a requirement for practical use.

Other recent works (Cuconasu et al., 2024; Liu et al., 2023; Kandpal et al.,
2023; Mallen et al., 2023) attempt to evaluate LMs and RAG systems in terms of
factual recall by exact matching of ground truth text in the generated text.
While this is a fast and efficient process, it faces serious challenges and
limitations in the verification task. As recognised by Cuconasu et al. (2024),
exact matching of ground truth text in the generated text is prone to false
negatives since the ground truth information might be present in the generated
text but phrased differently. Secondly, this approach cannot be applied when
the ground truth information longer than a few words since the chances of
exact match become too slim.

To address the challenges above, we make the following contributions:

  1. 1.

We propose an end-to-end factual recall evaluation framework which is tailored
to RAG systems. It can be used to (i) create a test dataset and (ii) perform
automated factual recall evaluation given a RAG system (or more loosely an
LM).

  2. 2.

We probe into the performance of fact verification formulations in conditions
of highly incomplete or inaccurate generated text. To do that, we augment
WikiEval111https://huggingface.co/datasets/
explodinggradients/WikiEval(Es et al., 2023) with ground truth facts and human
annotation of their validity. WikiEval features question/answer pairs with
answers of variable factual quality which enable simulating deficient RAG
responses. We find that prompt-based fact verification faces serious
challenges in identifying unsupported facts in the presence of inaccurate or
incomplete generated text.

  3. 3.

We introduce Facts as a Function (FaaF), a new fact verification formulation
which outperforms fact verification via prompting in all examined conditions
and reduces the required number of LM calls and completion tokens by more than
5 times.

  4. 4.

We open source the factual recall evaluation framework with FaaF as a python
package (pip install faaf) and the augmented WikiEval dataset to help the
community include factual recall in the RAG optimisation and ultimately build
more reliable systems.

##  2 Related Work

Recently, authors introduced RAGAS (Es et al., 2023), an evaluation framework
for RAG systems which is measuring the performance of retrieval and generation
without the requirement of ground truth human annotation. Although the authors
recognise and are motivated by the need for a practical an efficient RAG
evaluation method, RAGAS does not capture the factual aspect of the
evaluation, which is a key performance metric and the central focus of RAG’s
intended use.

Min et al. (2023) use prompt variations to directly verify individual facts
extracted from LM generated biographies. They compare their method with human
evaluation and find low error rates when retrieving the ground truth for the
evaluated fact.

Zhang et al. (2023) propose self-measuring factuality by the LM via a few-shot
prompting method combined by generated facts pertinent to the statement in
question. They argue that while leveraging facts from a knowledge base is more
dependable, its effectiveness is confined to the scope of the knowledge base
and the quality of retrieval. Conversely, self-evaluating with generated facts
offers more flexibility but risks introducing inaccuracies.

Li et al. (2023) indicate that LMs have difficulty identifying non-factual
information with standard prompting strategies and report improvement using
Chain of Thought (CoT).

Azaria and Mitchell (2023) also find that fact verification by prompting
insufficient and propose to train a classifier on the hidden layer activations
of open source LMs to predict the truthfulness of a generated statements.
However, current leading commercial models (GPT-4) are lacking layer
activation access and so require alternative methods.

Another approach in the same spirit is to look at the conditional
probabilities of each generated token as an indicator of LM confidence and
truthfulness of the generated text with the view that low LM confidence is a
proxy for incorrect statements (Yuan et al., 2021).

Fu et al. (2023) build on the concept of utilising token probabilities,
introducing a self-evaluation framework for LMs. This framework leverages few-
shot prompting to evaluate various instructed aspects of LM responses, such as
factuallity, fluency, interest, among others.

Manakul et al. (2023) propose SelfCheckGPT which automates the detection of
factual errors in LM outputs through statistical analysis of multiple
responses to the same prompt, without external knowledge sources. This is
again, an expression of the general idea that the probability distribution of
the generated response is indicative to the confidence on its truthfulness. So
similar to Yuan et al. (2021), SelfCheckGPT makes this assessment post LM-
generation by sampling multiple answers on the same prompt thereby removing
the requirement of access to the token probabilities or layer weights and
making this approach applicable to closed models.

Aly et al. (2021) use a Roberta encoder with a linear layer to learn and
predict the fact label given text evidence.

Wang et al. (2023) describe a method where the LM is prompted directly to
score an answer’s specific aspect from 0 to 100 or rates it on a 5-star scale,
yielding notable results. However, this approach’s effectiveness heavily
depends on the prompt’s design.

Zhang et al. (2020) attempt a flexible self-evaluation of generated text using
reference answers (BertScore). BertScore calculates a similarity score between
tokens in the generated and reference sentences using contextual embeddings.
The key benefit being that there is no reliance on exact matching between
generated and reference text. Nevertheless, a high semantic score at sentences
level does not guarantee factual precision, especially when the information
examined is not contextual and explicit (E.g. date).

The work of Zhao et al. (2019) also relies on contextual embeddings but their
approach allows for an intentional bias towards precision or recall via
reformulating the semantic similarity between generated and reference text as
an optimisation problem of finding the minimum effort to transform between the
two.

Kadavath et al. (2022) observing that LLMs offer well-calibrated probabilities
for multiple-choice questions, effectively transform the validation of model-
generated answers into a True/False multiple-choice question. This work offers
an important insight in that constraining the the LM response into a binary
option, it incites a sufficiently confident response which allows for self-
evaluation.

Lastly, recent work on the the factual accuracy of LMs and RAG systems
(Cuconasu et al., 2024; Liu et al., 2023; Kandpal et al., 2023; Mallen et al.,
2023) took the approach of using the NaturalQuestions-Open (NQ-
open)222https://ai.google.com/research/NaturalQuestions dataset (Kwiatkowski
et al., 2019) and calculate accuracy by judging whether any of the ground
truth answers (NaturalQuestions annotations) appear in the predicted output
via exact matching. NQ-open is a large scale dataset which comprises
historical Google search queries and their human-annotated answers sourced
from Wikipedia. Even though NQ-open, is valuable for its extensive scope and
domain-agnostic nature, fact-verification via exact matches faces serious
challenges. As highlighted by Cuconasu et al. (2024), a key issue is
accurately determining the truthfulness of answers, especially when dealing
with phrases that have the same meaning but (slightly different format or
wording) or different date formats. Thus, the necessity for a more advanced
analysis of answer variations is recognise and left for future research.

##  3 Facts as a Function

Facts as a Function (FaaF) introduces a streamlined fact verification method
using structured objects for multi-fact assessment. Although in this paper we
focus in examining the FaaF approach in the scope of factual recall evaluation
in RAG systems, the method itself can equivalently be used for broader fact
verification tasks.

Key idea 1: JSON over prompt
We propose that by using the function calling ability of the LM, we enforce a
more formal mode of reasoning and communication compared to natural language
for a number of reasons. First, by training on code and function definitions
the LM learns the independence of the function variables with each other, this
results in a clearer individual assessment of each requested variable. In
addition, by leveraging function variable metadata and type hints, the LM can
be better constrained to the accepted modes of response for each variable.
This provides a more consistent guidance compared to instructions given once
at the end or the beginning of a prompt. Building on the findings of Kadavath
et al. (2022), who established that LMs show well calibrated probabilities
when presented with multiple choice questions, we propose that using the
function’s type definition of the variables to convey the accepted LM
responses is a step further in the same direction, which aids the calibration
of the token probabilities. Lastly, due to the strict nature of code syntax
compared to natural language, gradients during training are expected to be
steeper and thus the LM learns to do a better job at adhering to the expected
output and respond with lower stochasticity - which is beneficial in the fact
verification use case.

Key idea 2: Generated text as a unit
As discussed above, a function definition can encapsulate a sufficient number
of variables to be used by the LM so that they capture all the facts
statements which need to be verified. Therefore, we move away from the concept
that each fact should be verified individually via a fact-specific prompt and
we propose instantiating a function per LM-generated text which needs to be
factually assessed. The function can be programatically generated in a manner
such that it includes all the individual facts which need to be verified in a
given piece of text. In other terms, function calling allows for the fact
verification of a long-form text as a unit, with a single LMeval call. This
approach results in a reduction of cost and time for fact-verification which
is multiple to the number of facts which would otherwise need to be assessed
individually, as seen in Chen et al. (2022); Gao et al. (2023); Min et al.
(2023); Lee et al. (2023).

Key idea 3: Outsourcing judgement from the LM to the function
Using function definitions to communicate with the LM, enables access to a
multitude of tools and further logic processing we can execute on LMeval’s
output. This strategy permits us to delegate certain deterministic judgments
away from LMeval. We demonstrate this capability by mapping a range of LM
responses into a binary format (True / False). In doing so, the calibration of
the LM response is enhanced as we provide a more accurate representation of
the spectrum of potential outcomes than a simple True/False dichotomy. The
underlying intuition is that ultimately, we can afford to ask LMeval simpler
and clearer questions which can be answered more reliably and further process
the LM output into a final response.

Definition
This work introduces the idea of presenting the facts to the LM as a callable
function. Let $S$ be the list of fact statements as strings to verify.

$S=[s_{1},s_{2},\ldots,s_{n}].$

A constructor function $C$ then maps the input list of facts $S$ and control
parameters $P$ to an object $O$ with fields $f$ and an object-specific parsing
function $G_{O}$.

$C(S,P)\to\left(O\left(f_{1},f_{2},\ldots,f_{n}\right),G_{O}\right).$

Each field from $(f_{1},f_{2},\ldots,f_{n})$ corresponds to a fact statement
and is further parameterised by $P$. Control parameters $P$ include the
methods and field properties which are injected into resulting object $O$. The
object-specific parsing function $G_{O}$ is tasked with parsing the LM
response in accordance to the schema and embedded instructions and methods
found in $O$. Before passing object $O$ to a language model, we convert it to
a JSON representation:

$J(O)\to\text{JSON}_{O}$

Let $M$ be the language model used for fact verification (LMeval). The input
of $M$ is a concatenation of $JSON_{O}$, a prompt $q$ which instructs $M$ to
utilise $O$ and the input text $x$ which is to be assessed for factual recall
with respect to the given facts $S$.

$M(\text{JSON}_{O},q,x)\to o_{x}$

$M$ responds with the output $o_{x}$ which is of string type. Then, the
parsing function $G_{O}$ is used to convert the raw response $o_{x}$ to a
callable function object $O^{\prime}$ which is an instance with values
assigned on its fields.

$G_{O}(o_{x})\to\acute{O}$

Finally, the fact-verification process can be expressed as

$\acute{O}=G_{O}(M(\text{JSON}_{O},q,x)).$

Figure 2: Overview of the factual recall evaluation for RAG. Given a set of
ground truth Answers, facts are extracted via LMf. The Hypothesized responses
of the RAG (in this instance Ungrounded Answer and Poor Answer) are then
tested for recall against the extracted facts.

##  4 Assessment of fact verification formulations in the RAG setting

Figure 2 outlines the the factual recall evaluation framework which also
serves as the experimental setup which we use to compare fact verification
formulations with each other. Starting from the ground truth answer containing
the desired information to fully answer the posed question, we derive a set of
fact statements using a fact-generator LM (LMf). We then use these derived
facts to evaluate each of the other answer variants of WikiEval for their
factual recall via LMeval. In this manner, each answer is evaluated with
respect to the information that is expected from it. It is easy to see how
this framework could be applied in the RAG setting where different
configurations or model choices impact the quality of the final response to
the same input question and knowledge base.

Dataset

In order to probe into performance of automatic fact verification methods we
chose to work with the WikiEval dataset (Es et al., 2023) which features three
versions of answers, of variable quality, to questions on Wikipedia articles.
Specifically, for each question, there is an answer (referred to as ground
truth answer in this text for clarity), ungrounded answer and poor answer. All
answers have been generated with GPT-3.5-turbo-16k.

The ground truth answer is generated by providing the LM with the correct
context from the respective Wikipedia page. The ungrounded answer is generated
by not providing any context to the LM. Finally, the poor answer is generated
by instructing the LM to give an incomplete answer to the passed question. An
example of each answer type can be seen in Figure 2. Generally, ungrounded
answer answers feature false, incomplete and redundant information with
respect to the ground truth answers. On the other hand,  poor answers contain
mostly incomplete information compared to ground truth answers i.e. no
evidence for support or rejection for ground truth facts.

This dataset enables us to focus on the performance of the fact-verification
as a function of the quality and completeness of different answer variants. In
that way, we test closely the ability of different fact verification methods
to identify unsupported facts when presented with (i) incorrect, (ii)
indirectly relevant and (ii) incomplete information with some degree of
distinction.

Fact generation

To prepare the WikiEval dataset, we initially generate factual statements that
fully capture the information from the ground truth answers, followed by
manual annotation of a version of these facts for each answer variant. We use
gpt-4-0125 to generate facts from the questions and ground truth answers from
the WikiEval dataset (QA pair) using the following prompt:

> Convert the given passage into a list of short facts which specifically
> answer the given question.
>
> Make sure that the facts can be found in the given passage.
>
> The facts should be coherent and succinct sentences with clear and simple
> syntax.
>
> Do not use pronouns as the subject or object in the syntax of each fact.
>
> The facts should be independent to each other.
>
> Do not create facts from the passage which are not answering the given
> question.
>
> Add a ”-” before each fact.
>
> Passage: [ground truth answer]
>
> Question: [question]

Fact generation via the prompt above results in a variable number of facts for
each ground truth QA pair which depends on the length and information density
in the processed ground truth answer. This process yielded 281 individual
facts for each type of answer (thus 843 facts in total considering answer,
ungrounded answer and poor answer) with an average of 5.6 fact statements
generated for every QA pair. The prompt has been designed to ensure that the
generated facts are complete sentences, understandable independently of each
other or any external references.

Answer Type | Human Accuracy
---|---
Answer | 100
Ungrounded Answer | 30.24
Poor Answer | 8.53
Table 1: Factual accuracy of the facts derived from the ground truth Answer of
WikiEval from human evaluation.

Human fact-verification

We outsource the fact verification of the generated facts against the triplet
of answers in WikiEval (answer, ungrounded answer and poor answer) to human
evaluators. In this manner we build a ground truth evaluation for each answer
type, enabling us to assess the effectiveness of automated fact-verification
methods against it. The accuracy from the human fact verification can be seen
in Table 1 where the deterioration of the ungrounded answer and poor answer
relative to the ground truth answer is evident.

Prompt fact-verification
Following Min et al. (2023), we use a prompt and the respective answer variant
as context to verify a single fact at a time with LMeval:

> Passage: [answer]
>
> Considering the given passage, the claim [fact] is True or False?

Facts as a function

For each set of fact statements which encapsulate a ground truth answer, we
construct a facts-specific object which also contains a parsing method. Since
the created object contains all the input facts as fields, we essentially
process the set of facts as a unit. A sample of the JSON representation of a
function object containing the first fact can be seen bellow:

    {’properties’:
        ‘fact_0:{’description’:"Respond
        to the claim in the ’title’ by
        using one of the accepted Enum
        types",
       Ψ’enum’: [’True’, ’False’],
       Ψ’title’: ’It is clear in
        the given passage that Pope
        Benedict XVI became the head
        of the Catholic Church and
        sovereign of the Vatican
        City State on April 19, 2005’
       Ψ’type’: ‘string’},
    …

     },
    ’required’:
    Ψ[‘fact_0’,
     Ψ‘fact_1’
     Ψ …,
     Ψ‘fact_n’]
    ’title’: ’FactChecker’,
    ’type’: ‘object’}

Each field includes metadata which can be used to pass instructions, type
annotations and the fact statement to be verified itself. In addition to the
function object, we pass the following prompt:

> Consider the given passage and assign the correct values in the fact checker
> function.
>
> Passage: [answer]

The answer in the prompt is the input text we want to evaluate against the
respective facts which have been previously derived by the ground truth answer
in our dataset. After LMeval generates a response, we process it using the
tailored parsing method previously created and instantiate the function object
that was passed into LMeval, populating it with the generated values in their
designated fields.

We test the following configurations:
FaaF(T/F)
A function object with fields which only accept True or False as a response
from LMeval.
FaaF(T/F/N)
A function object with fields which only accept True, False or Not clear from
the given passage as a response from LMeval. In this scenario, the parser will
map Not clear from the given passage to False when processing the LMeval
output. This is an example of applying logic post LM generation. The intuition
behind this configuration is that the rejection of a claim based on
contradicting evidence is conceptually different to the rejection of a claim
based on absence of evidence and we help the LMeval’s calibration by providing
a clear option for each.
FaaF(T/F)+citation
In this instance we construct a function object with two fields for each input
fact. One field for the factual evaluation and one field where we instruct the
LMeval to generate an exact excerpt from the input text which directly
supports the fact in question (citation field). We place the citation field
prior to the factual evaluation field so that LMeval is made to first try and
find a supporting citation from the input text before verifying the fact that
is being assessed. Similarly, the intuition here is that by asking LMeval to
search and retrieve a specific citation from the input text which supports a
specific fact, it will result in a better calibrated verification of the
respective fact statement.
FaaF(T/F/N)+citation
In this configuration we combine the two approaches outlined above to explore
their combined effect. In detail, we construct the function object to include
citation fields and factual evaluation fields for each fact and we define
True, False or Not clear from the given passage as accepted responses.

Language models (LMeval)
We use the latest commercially available model which support function calling
gpt-4-0125. A comprehensive assessment of open source models which support
function calling is left for future study as there are none available at this
time but we will extend this study to more models as they become available.
Also, the focus of this work is on the different formulation of facts during
verification and not the assessment of specific LMs.

Metrics
Error rate
We use Error Rate (ER) between the human fact-verification and the fact
verification formulation as the main indicator of their reliability.
F1micro
We also use F1micro score (F1m) as defined in Min et al. (2023) to measure the
successful identification of unsupported facts and probe further into the
individual fact verification. It should be noted that F1 scores explicitly
depend on the class ratio (T/F) via precision and recall. For that reason F1m
scores should be compared across fact verification approaches in the same
answer category (where the T/F ratio is preserved) in Table 2, and not across
answer categories.

Facts Formulation | (ground truth) Answer | Ungrounded Answer | Poor Answer
---|---|---|---
| N/A | ER | F1m | N/A | ER | F1m | N/A | ER | F1m
Prompt(T/F) | 0/281 | 2.1 | 0∗ | 4/281 | 26.7 | 77.6 | 3/281 | 50.8 | 61.8
FaaF(T/F) | 0/281 | 1.7 | 0∗ | 0/281 | 24.9 | 79.6 | 0/281 | 9.9 | 94.2
FaaF(T/F/N) | 0/281 | 1.4 | 0∗ | 0/281 | 18.1 | 85.7 | 0/281 | 7.4 | 95.7
FaaF(T/F)+citation | 0/281 | 0.7 | 0∗ | 0/281 | 22 | 82.5 | 0/281 | 7.8 | 95.6
FaaF(T/F/N)+citation | 0/281 | 0.3 | 0∗ | 0/281 | 23.1 | 81.7 | 0/281 | 7.4 | 95.8
Table 2: Results on Error Rate (ER) along with F1micro estimated by each fact
formulation. F1micro measures accuracy and recall in predicting the false
facts. ∗F1micro is 0 in the Ground Truth Answer cases because there are no
False facts in the ground truth evaluation in this category. T/F indicates
True/False and T/F/N are True/False/Not clear from the given passage
respectively. N/A indicates the Not answered i.e. the number of facts that the
LM did not respond on their validity.

##  5 Results

Table 2 presents the non-response rate (N/A), error rate and F1m from the fact
verification formulations across the answer categories which are examined.

Prompting for verification is not reliable in cases of incorrect or incomplete
information. Although in the case of ground truth answers Prompt(T/F) shows
2.1%ER (in line with what is reported in Min et al. (2023) when retrieval is
enabled), we see a sharp rise when we try to verify facts in text from the
ungrounded answers (26.7%) and a further deterioration with poor answers
(50.8%). Such performance indicates that prompting is not suitable for fact
verification in text with unknown information quality and completeness.

FaaF outperforms zero shot prompt in all cases. Formulations which leverage
function calling demonstrate a notable improvement in ER and F1micro over all
cases (answer type, FaaF configuration). The improvement is most pronounced in
the poor answer category where ER is reduced from 50.8% (prompting) to less
than 10% with FaaF methods denoting a paradigm shift in LMeval’s ability to
identify missing information in the input text. The milder improvement seen in
the ungrounded answer category suggests that elucidating the fact verification
in the presence of incorrect and indirectly relevant information is more
challenging for LMeval. Lastly, in the case of prompting, LMeval failed to
respond on 4/281 of the examined facts for the ungrounded answer and 3/281 for
the poor answer. Conversely, LMeval returned a valid response for all facts,
in all FaaF configurations and answer types.

LMs tend to overestimate fact truthfulness overall. The human evaluation of
factual accuracy in ground truth answers is 100% i.e. every fact is True
(Table 1). This coincides with the lowest ER scores in Table 2, irrespective
of the fact verification approach.

Rejecting facts is more difficult in the presence of tangentially relevant
information when using FaaF. It is evident from Table 2 that FaaF approaches
struggle more with ungrounded answers than with poor answers. It should be
noted that this observation is reversed when using prompt(T/F), where the ER
is higher in poor answers, due to the significant overestimation of
truthfulness this approach shows in the absence of supporting information. The
key difference between the two answer types is that ungrounded answers
typically contain incorrect or tangentially relevant information to the facts
that are examined whilst poor answers usually lack information altogether.
This is because ungrounded answers are generated by an LM without any context
thus, depending on the popularity and prevalence of the theme of the question,
a degree of factual hallucination can be expected.

Consider the following example from WikiEval:

Question:

> What factors contributed to the Sri Lankan economic crisis?

Ground truth answer:

> The Sri Lankan economic crisis was caused by multiple compounding factors.
> These include tax cuts and money creation by the government, which led to
> budget deficits and a decline in government revenue. The government’s
> decision to shift to organic or biological farming also played a role.
> Additionally, the crisis was exacerbated by the 2019 Sri Lanka Easter
> bombings and the impact of the COVID-19 pandemic. The country’s high
> external debt, with a significant increase in foreign debt in recent years,
> also contributed to the crisis. The combination of these factors resulted in
> unprecedented levels of inflation, near-depletion of foreign exchange
> reserves, shortages of medical supplies, and an increase in prices of basic
> commodities.

Ungrounded answer:

> There were several factors that contributed to the Sri Lankan economic
> crisis:
>
>   1. 1.
>
> High levels of government debt: Sri Lanka had accumulated a significant
> amount of government debt, both domestic and foreign. This debt burden
> became unsustainable, leading to a crisis in public finances.
>
>   2. 2.
>
> Fiscal mismanagement: The government had been running large budget deficits
> for several years, spending more than it was earning. This led to a widening
> fiscal deficit and increased borrowing to finance government expenditure.
>
>   3. 3.
>
> Political instability: Sri Lanka experienced political instability during
> this period, with frequent changes in government and policy uncertainty.
> This created an unfavourable business environment and deterred foreign
> investment.
>
>   4. 4.
>
> Declining exports: Sri Lanka’s export sector, particularly garments and
> textiles, faced challenges due to global economic slowdowns and increased
> competition from other countries. This led to a decline in export earnings,
> which further strained the country’s external finances.
>
>   5. 5.
>
> High inflation: Sri Lanka experienced high inflation during this period,
> which eroded the purchasing power of the population and increased the cost
> of living. This put additional pressure on the economy and contributed to
> the crisis.
>
>   6. 6.
>
> External shocks: Sri Lanka was also affected by external shocks, such as
> rising oil prices and global financial market volatility. These factors
> further exacerbated the economic challenges faced by the country.
>
>   7. 7.
>
> Structural issues: Sri Lanka had several structural issues that hindered its
> economic growth, such as a lack of diversification in the economy,
> inadequate infrastructure, and low productivity levels. These issues made it
> difficult for the country to sustain economic growth and overcome the
> crisis.
>
>

Poor answer:

> There were several factors that contributed to the Sri Lankan economic
> crisis.Sri Lanka is known for its beautiful beaches and rich cultural
> heritage.

Extracted facts(using question and ground truth Answer):

>   * •
>
> Tax cuts and money creation by the government contributed to the Sri Lankan
> economic crisis.
>
>   * •
>
> Budget deficits and a decline in government revenue were factors in the Sri
> Lankan economic crisis.
>
>   * •
>
> The governments decision to shift to organic or biological farming played a
> role in the crisis.
>
>   * •
>
> The 2019 Sri Lanka Easter bombings exacerbated the economic crisis.
>
>   * •
>
> The impact of the COVID 19 pandemic contributed to the Sri Lankan economic
> crisis.
>
>   * •
>
> High external debt, with a significant increase in foreign debt in recent
> years, also contributed to the crisis.
>
>

Although the ungrounded answer is quite verbose and has several mentions and
indirect references of the extracted facts, it fails to capture with clarity
the information from the ground truth answer which would allow for their
confident verification. Meanwhile, the poor answer caries no useful
information in this instance.

In this scenario LMeval has a higher risk of a misjudgement in the ungrounded
answer than the poor answer. This seems coherent intuitively since rejecting a
claim in the presence of relevant information is a more demanding and complex
task which requires deeper interpretation of the language than when there is
no relevant information.

Figure 3: LMeval call count for a full evaluation of WikiEval. FaaF formulations result in more than five times less LM calls. Facts formulation | Prompt Tokens | Completion Tokens | Total Tokens
---|---|---|---
Prompt(T/F) | 146,276 | 26,545 | 172,821
FaaF(T/F) | 69,797 | 5,658 | 75,455
FaaF(T/F/N) | 77,384 | 7,113 | 84,497
FaaF(T/F)+citation | 122,600 | 20,596 | 143,196
FaaF(T/F/N)+citation | 130,187 | 21,889 | 152,076
Table 3: LMeval token count for the tested fact formulations, for the factual
recall evaluation of WikiEval dataset.

Providing a “not clear” option helps. We observe a reduction of the error rate
and corresponding increase in F1m when we add the option for LMeval to respond
with Not clear from the given passage which is mapped to False as a post-
generation step in the parsing function. The helpful mechanism in this
instance is that we provide a needed third option to LMeval when the token
probability distribution between True / False is not clearly indicating one
over the other. Rejecting a statement due to conflicting evidence and due to
lack of evidence are both as valid reason as they are distinct to each other.
Using False to capture both rejection scenarios has proved to lead to more
false positives than providing LMeval with the option to distinguish between
them.

Citations help only in the presence of correct and clear information. The
positive impact of adding citations is most evident in the ground truth answer
category where the provided text always contain the required evidence to
support the facts. In this instance, asking for text evidence from LMeval
results in avoiding some false negative verifications. The beneficial
mechanism is associated with inserting a preliminary step to the fact
verification process - of the explicit use of evidence from the input text.
This aligns with the findings reported in Wei et al. (2023) regarding the
chain-of-thought method.

Interestingly, for the other answer categories, the citation benefit becomes
less clear and even reversed. ER is relatively stable in poor answers and it
is seen to increase in the case of ungrounded answer. In detail, we notice two
conflicting trends: firstly, citations can prevent false positives by
highlighting the absence of supporting text for a given fact statement.
Conversely, in a few instances, the lack of explicit citations leads to false
negatives, as LMeval fails to recognize a fact as true, even though a holistic
human interpretation of the answer would deem it accurate.

LMeval usage efficiency

Beyond the improvements in ER and F1m, another advantage in encapsulating
multiple facts into a function object is the that of efficiency. Figure 3
shows reduction in the number of calls to LMeval required for the factual
evaluation of WikiEval by more than five times. The number of calls to LMeval
required for FaaF corresponds to the number of texts to be evaluated, whereas,
in prompt-based fact verification, it matches the number of fact statements.

Table 3 indicates that FaaF(T/F) uses half the prompt tokens and less than
one-fifth of the completion tokens required by prompt-based fact verification
for a complete evaluation on WikiEval(all answer types). Particularly in the
setting of RAG systems optimisation, the efficiency and scalability of an
evaluation framework is of high importance since it is proportionally related
to the number of RAG configurations which can be tested.

Including citations and the option Not clear from the given passage as
acceptable responses in FaaF affects the token count, significantly increasing
it. However, it remains below the token requirements of prompt-based fact
verification. In essence, FaaF makes possible fact verification with citations
at a lower token cost than prompt-based verification without citations.

##  6 Conclusions & future work

We show that prompt-based fact verification is prone to overestimating the
truthfulness of fact statements in texts with inaccurate and/or missing
information. In particular the error rate of prompting can reach up to 50%
when the text under review is significantly lacking in information. In such
challenging situations, presenting the facts as a function to the LM
significantly enhances both the error rate and the F1m measure of the
verification process. Using FaaF, we observe that texts with tangentially
relevant and inaccurate information’s are more likely to cause false positives
than texts with missing or incomplete information. By testing various
configurations with FaaF, we find that by including a “not clear” option to
the True/False dichotomy helps overall. The impact of asking for citations
before fact verification is sensitive to the quality and coverage of
information in the examined text. While beneficial when fact-supportive
information is present or clearly absent, this approach can lead to false
negatives if the text can support the fact as a whole but lacks direct,
extractable citations.

Finally we report significant cost and time efficiency improvements between
prompting and function based fact representations. Generally, using FaaF leads
to a reduction in both the number of calls to the LM and the number of tokens
needed for fact verification by a multiple-factor.

## Limitations

While the advantages of using function calls for fact verification are
significant, there is certain variance expected in the results due to the
stochastic nature of LM generation. Further extensive testing is necessary to
solidify the results presented, especially probing into the influence of
citations in the performance of verification using function objects. Although,
the WikiEval dataset has been instrumental in providing a convenient way to
compare fact verification performance across various text qualities, it is
relatively small, comprising only 50 question/answer pairs.

Additionally, the results shown with FaaF are sensitive to the instructions
passed in the function object’s metadata, which highlights the need for
additional research into different FaaF configurations. Other open questions
include the maximum number of fact statements and the maximum permissible
length for a fact that can be incorporated into a function object and whether
token count is the sole limitation or if there are performance implications as
well.

## Acknowledgements

This research was supported by IMMO Capital, London UK.

## References

  * Aly et al. (2021) Rami Aly, Zhijiang Guo, Michael Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, and Arpit Mittal. 2021.  Feverous: Fact extraction and verification over unstructured and structured information.
  * Azaria and Mitchell (2023) Amos Azaria and Tom Mitchell. 2023.  The internal state of an llm knows when it’s lying.
  * Chen et al. (2022) Jifan Chen, Aniruddh Sriram, Eunsol Choi, and Greg Durrett. 2022.  Generating literal and implied subquestions to fact-check complex claims.
  * Cuconasu et al. (2024) Florin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. 2024.  The power of noise: Redefining retrieval for rag systems.
  * Es et al. (2023) Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. 2023.  Ragas: Automated evaluation of retrieval augmented generation.
  * Fu et al. (2023) Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023.  Gptscore: Evaluate as you desire.
  * Gao et al. (2023) Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Y. Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu. 2023.  Rarr: Researching and revising what language models say, using language models.
  * Kadavath et al. (2022) Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. 2022.  Language models (mostly) know what they know.
  * Kandpal et al. (2023) Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023.  Large language models struggle to learn long-tail knowledge.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural questions: A benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:452–466.
  * Lee et al. (2019) Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019.  Latent retrieval for weakly supervised open domain question answering.
  * Lee et al. (2023) Nayeon Lee, Wei Ping, Peng Xu, Mostofa Patwary, Pascale Fung, Mohammad Shoeybi, and Bryan Catanzaro. 2023.  Factuality enhanced language models for open-ended text generation.
  * Li et al. (2023) Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2023.  Halueval: A large-scale hallucination evaluation benchmark for large language models.
  * Liu et al. (2023) Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023.  Lost in the middle: How language models use long contexts.
  * Mallen et al. (2023) Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023.  When not to trust language models: Investigating effectiveness of parametric and non-parametric memories.
  * Manakul et al. (2023) Potsawee Manakul, Adian Liusie, and Mark J. F. Gales. 2023.  Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models.
  * Min et al. (2023) Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023.  Factscore: Fine-grained atomic evaluation of factual precision in long form text generation.
  * Wang et al. (2023) Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023.  Is chatgpt a good nlg evaluator? a preliminary study.
  * Wei et al. (2023) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023.  Chain-of-thought prompting elicits reasoning in large language models.
  * Yuan et al. (2021) Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021.  Bartscore: Evaluating generated text as text generation.
  * Zhang et al. (2023) Tianhua Zhang, Hongyin Luo, Yung-Sung Chuang, Wei Fang, Luc Gaitskell, Thomas Hartvigsen, Xixin Wu, Danny Fox, Helen Meng, and James Glass. 2023.  Interpretable unified language checking.
  * Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020.  Bertscore: Evaluating text generation with bert.
  * Zhao et al. (2019) Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019.  Moverscore: Text generation evaluating with contextualized embeddings and earth mover distance.

’


# Spacetime surgery for black hole fireworks

Wei-Chen<EMAIL_ADDRESS>Dong-han
<EMAIL_ADDRESS>and Dejan<EMAIL_ADDRESS>aCenter for Cosmological Constant Problem, Extreme Physics Institute, Pusan
National University, Busan 46241, Republic of Korea
bDepartment of Physics, Pusan National University, Busan 46241, Republic of
Korea
cHEPCOS, Department of Physics, SUNY at Buffalo, Buffalo, NY 14260, USA
dDepartment of Physics Education, Pusan National University, Busan 46241,
Republic of Korea
eResearch Center for Dielectric and Advanced Matter Physics, Pusan National
University, Busan 46241, Republic of Korea
fLeung Center for Cosmology and Particle Astrophysics, National Taiwan
University, Taipei 10617, Taiwan

###### Abstract

We construct an explicit model for the black hole to white hole transition
(known as the black hole fireworks scenario) using the cut-and-paste
technique. We model a black hole collapse using the evolution of a time-like
shell in the background of the loop quantum gravity inspired metric. We then
use the space-like shell analysis to construct the firework geometry. Our
simple and well defined analysis removes some subtle issues that were present
in the previous literature. In particular, we demonstrate that the null energy
condition must be violated for the bounce. We also calculate the proper time
scales required for the black to white hole transition, which in any valid
scenario must be shorter than the evaporation time scale. In contrast, we show
that the bouncing time for the distant observer can be chosen arbitrarily,
since it is determined by how one cuts and pastes the spacetimes outside the
event horizon, and thus does not have any obvious connection to quantum
gravity effects.

###### Contents

  1. I Introduction
  2. II Bouncing black hole model
  3. III Time-like thin-shells and gravitational collapses
    1. III.1 Junction equations
    2. III.2 Analysis of the solution
  4. IV Space-like thin-shells and black hole fireworks
    1. IV.1 Junction equations
    2. IV.2 Conditions for thin-shells
    3. IV.3 Construction of the black hole firework geometry
  5. V Bouncing time-scale for black hole firework scenarios
    1. V.1 Bouncing time scale with the $\delta$ parameter
    2. V.2 Bouncing time for the comoving observer
    3. V.3 Interpretation: coordinate time and proper time
  6. VI Discussion

##  I Introduction

The issue of formation and evaporation of a black hole is very important for
understanding the nature of quantum gravity. In particular, this issue is
related to the information loss problem of an evaporating black hole
Hawking:1976ra . Is there a unitary theory of quantum gravity that explains
the unitary evolution of evaporating black holes? If there is, is this theory
consistent with the semi-classical description Yeom:2009zp ? Will the
classical singularity survive in the regime where quantum gravitational
effects are dominant Bouhmadi-Lopez:2019kkt ; Bogojevic:1998ma ?

It is clear that understanding the fate of the singularity is very important
to obtain the complete answer to the black hole evaporation and the
information loss problem. Intuitively, we may classify two ways. First, we may
address this problem by introducing a wave function, i.e., by solving the
Wheeler-DeWitt equation DeWitt:1967yk . In this approach, we need to solve the
Wheeler-DeWitt equation (or some version of it) and interpret the solution in
the classical background, which is sometimes a subtle problem Brahma:2021xjy ;
Saini:2014qpa ; Greenwood:2008ht ; Wang:2009ay ; for an attempt to model
quantum radiation from quantum background, see Vachaspati:2007hr . Second, we
may remove the singularity by introducing an effective matter Ayon-
Beato:1999kuh ; Nicolini:2005vd ; Frolov:1988vj ; Yeom:2008qw ; Bouhmadi-
Lopez:2021zwt . As a result, one could extend the effectively classical
spacetime beyond the singularity. However, we need to justify the ad hoc
introduced matter from the first principles, which is usually a difficult
task.

Interestingly, an approach coming from the loop quantum gravity provides a
method that is in between these two ways. In that approach, one first needs to
solve the Wheeler-DeWitt equation in order to obtain the physical quantum
state of the singularity. Usually, it is not easy to solve the Wheeler-DeWitt
equation directly. However, one may reasonably expect an effective
modification of the Hamiltonian which includes loop quantum gravitational
effects Bojowald:2018xxu . With this modified Hamiltonian, one can solve a set
of semi-classical equations and obtain a spacetime that includes loop quantum
gravitational effects, e.g., resolution of the singularity.

A typical solution in the framework of the loop quantum gravity includes
bouncing of the collapsing object Ashtekar:2018lag ; Ashtekar:2018cay .
Bouncing inside the horizon is not a very surprising scenario, except for some
technical issues Bouhmadi-Lopez:2019hpp . However, in reality, this is not
easy to generalize to global spacetimes in an evaporating background. In some
cases, inconsistencies may arise Hong:2022thd . In an evaporating background,
the bouncing spacetimes have to consistently connect not only inside but also
outside the horizon Brahma:2021xjy ; Ashtekar:2005cj . This might be realized
by cutting and pasting spacetimes, e.g. like in the Haggard-Rovelli model
Haggard:2014rza . Moreover, if we modify the interior of the black hole
solution, one can obtain a bouncing model that has two horizons Han:2023wxg .
The scenario proposed in Haggard:2014rza ; Han:2023wxg  is also known as the
black hole fireworks.

One can revisit the cut-and-paste technique of Haggard:2014rza  and
Han:2023wxg  using the thin-shell approximation Israel:1966rt . This spacetime
surgery could explain the global spacetime of the Haggard-Rovelli model
Brahma:2018cgr . For this purpose, one needs to introduce a space-like shell
and paste two space-like hypersurfaces. To do this in a self-consistent way, a
space-like matter shell that violates the null energy condition and reaches
asymptotic infinity is required.

In this paper, we further extend this idea to the model in Han:2023wxg , which
contains two horizons. We consider a time-like shell that describes a
collapsing star interior and the dynamical formation of a black hole. In
addition, we cut and paste two spacetimes to accommodate a bouncing spacetime,
and also cover both the outer and inner apparent horizons. This approach is
technically well-defined and hence allows a more concrete and reliable way to
evaluate the transition time scale from the collapsing to the bouncing phase.
Unless this time scale is sufficiently long, this process will be already
excluded by astrophysical observations.

This paper is organized as follows. In Sec. II, we describe the black hole
bouncing model of Han:2023wxg  in which a black hole phase is followed by a
white hole phase. In Sec. III, we consider a collapsing time-like shell and
the dynamical process of black hole formation. In Sec. IV, we consider a
space-like shell that separates the black hole and white hole phases in a cut-
and-paste procedure. In Sec. V, we discuss the bouncing time scales from the
black hole and white hole phases. Finally, in Sec. VI, we summarize our
results and discuss possible future research.

##  II Bouncing black hole model

We consider the black hole model defined inHan:2023wxg , which has a quantum-
corrected center. The metric is

$\displaystyle ds^{2}=-f(r)dt^{2}+\frac{1}{f(r)}dr^{2}+r^{2}d\Omega^{2},$ (1)

with

$\displaystyle f(r)=1-\frac{2M}{r}+\frac{AM^{2}}{r^{4}},$ (2)

where $M$ is the black hole mass, while $A$ is a constant. Generically, this
geometry has two horizons, labeled with $r_{\pm}$, with a time-like center
(Fig. 2). We note here that the bounce in this model is driven by the
$AM^{2}/r^{4}$ term in Eq. (2). This term dominates only at small values of
$r$, and provides a repulsive gravity. Thus, directly from this form, we
expect an oscillating behavior. At large values of $r$, the attractive term,
$2M/r$, dominates and drives the collapse. At some minimal value of $r$, the
repulsive term causes bounce and pushes the collapsing object out to larger
values of $r$ where the attractive term again dominates and the cycle starts
again.

Figure 1: The Penrose diagram of the model in Eq. (1). The solution has two
horizons, $r_{\pm}$, and the time-like center.

Figure 2: Dynamics of the time-like shell. Top left: $V_{\mathrm{eff}}$ with
$M=10$, $A=0.1$, and $\sigma_{0}=0.04$. There are two bouncing points located
at $r_{\mathrm{max}}\simeq 19.9993$ and $r_{\mathrm{min}}\simeq 0.795$. Note
that the outer horizon is $r_{+}=19.9987$ and the inner horizon is
$r_{-}=0.8046$. Top right: $V_{\mathrm{eff}}$ around $r_{\mathrm{min}}\simeq
0.795$. Bottom: $\beta_{+}$ (black) and $\beta_{-}$ (red). This shows that for
$r_{\mathrm{min}}\leq r\leq r_{\mathrm{max}}$, $\beta_{\pm}>0$ conditions are
satisfied.

##  III Time-like thin-shells and gravitational collapses

We first consider a thin time-like shell in order to explain the process of
gravitational collapse in the framework of the model given in Eq. (1).

###  III.1 Junction equations

The metric outside and inside the shell is

$\displaystyle
ds_{\pm}^{2}=-f_{\pm}(r)dt^{2}+\frac{1}{f_{\pm}(r)}dr^{2}+r^{2}d\Omega^{2},$
(3)

where $+$ and $-$ stand for outside and inside the shell, respectively. The
metric of the time-like shell is

$\displaystyle ds_{\mathrm{shell}}^{2}=-d\tau^{2}+r^{2}(\tau)d\Omega^{2}.$ (4)

Here, we assume $f_{+}=f(r)$ and $f_{-}=1$.

After imposing the junction equation Israel:1966rt , we obtain

$\displaystyle\epsilon_{-}\sqrt{\dot{r}^{2}+f_{-}}-\epsilon_{+}\sqrt{\dot{r}^{2}+f_{+}}=4\pi
r\sigma(r),$ (5)

where $\sigma(r)$ is the tension of the shell, and $\epsilon_{\pm}=\pm 1$ are
the signs of the extrinsic curvatures. Here, extrinsic curvatures
$\beta_{\pm}$ are

$\displaystyle\beta_{\pm}\equiv\frac{f_{-}-f_{+}\mp
16\pi^{2}\sigma^{2}r^{2}}{8\pi\sigma
r}=\epsilon_{\pm}\sqrt{\dot{r}^{2}+f_{\pm}}.$ (6)

Note that if $\epsilon_{\pm}=+1$, $r$ increases along the outward normal
direction, while if $\epsilon_{\pm}=-1$, $r$ decreases along the outward
normal direction. Therefore, we have to assume $\epsilon_{\pm}=1$.

After simple computations, we obtain the equation:

$\displaystyle\dot{r}^{2}+V_{\mathrm{eff}}(r)=0,$ (7)

where

$\displaystyle
V_{\mathrm{eff}}(r)=f_{+}-\frac{\left(f_{-}-f_{+}-16\pi^{2}\sigma^{2}r^{2}\right)^{2}}{64\pi^{2}\sigma^{2}r^{2}}.$
(8)

Here, we interpret that $V_{\mathrm{eff}}<0$ corresponds to the region where
classical trajectories are allowed.

###  III.2 Analysis of the solution

In order to form a black hole, one can set $\sigma=\sigma_{0}$, and assume
$\lambda/\sigma=-1$, where $\lambda$ is the pressure of the shell. For
example, a scalar field can satisfy such a condition Blau:1986cw .

Fig. 2 is an example that describes the gravitational collapse of a time-like
shell and the formation of a black hole. Top left and right of Fig. 2 are
$V_{\mathrm{eff}}$, where we choose $M=10$, $A=0.1$, and $\sigma_{0}=0.04$.
For these values of parameters, $r_{+}=19.9987$ and $r_{-}=0.8046$. By
evaluating $V_{\mathrm{eff}}$, we find two bouncing points
$r_{\mathrm{max}}\simeq 19.9993$ and $r_{\mathrm{min}}\simeq 0.795$.
Therefore, $r_{\mathrm{min}}<r_{-}$ and $r_{\mathrm{max}}>r_{+}$, and hence,
the shell propagates from the region outside of the outer horizon to the
region inside of the inner horizon. In addition, bottom of Fig. 2 shows
$\beta_{+}$ (black) and $\beta_{-}$ (red), which indicates that for a
classically allowed region $r_{\mathrm{min}}\leq r\leq r_{\mathrm{max}}$, the
extrinsic curvatures $\beta_{\pm}$ are always positive, as we expected.

If we summarize these numerical results, one can conceptually reconstruct Fig.
3 as a Penrose diagram. The time-like shell is located between
$r_{\mathrm{min}}\leq r\leq r_{\mathrm{max}}$, where $r_{\mathrm{max}}$ is
outside the outer horizon and $r_{\mathrm{min}}$ is inside the inner horizon.
Using the cut-and-paste technique, we paste a Minkowski space inside the
shell. On the right side of the Fig. 3, there are dashed curves. These curves
apparently do not follow the thin-shell trajectories. However, assuming some
properties of a star interior, it is reasonable to assume that such a
stationary shell is located outside the horizon Chen:2015lbp .

Figure 3: Left: The Penrose diagram of the black hole solution, where the red
lines are outer horizons and the blue lines are inner horizons. There exists a
time-like shell solution that is oscillating between $r_{\mathrm{min}}\leq
r\leq r_{\mathrm{max}}$, where $r_{\mathrm{max}}$ is outside the outer horizon
and $r_{\mathrm{min}}$ is inside the inner horizon. Right: Inside the shell,
the geometry is Minkowski. This diagram represents the resulting spacetime of
the black hole formation.

##  IV Space-like thin-shells and black hole fireworks

To consider the black hole firework scenario, we need to cut and paste on top
of Fig. 3. We will introduce a space-like shell and use it to paste two space-
like slices Brahma:2018cgr .

###  IV.1 Junction equations

The metric outside and inside the shell:

$\displaystyle
ds_{\pm}^{2}=-\frac{1}{\tilde{f}_{\pm}(r)}dr^{2}+\tilde{f}_{\pm}(r)dt^{2}+r^{2}d\Omega^{2},$
(9)

where $+$ and $-$ denote outside and inside the shell. The metric of the
space-like shell is

$\displaystyle ds_{\mathrm{shell}}^{2}=ds^{2}+r^{2}(s)d\Omega^{2}.$ (10)

Here, we impose that

$\displaystyle\tilde{f}_{\pm}(r)=-f(r)=-1+\frac{2M}{r}-\frac{AM^{2}}{r^{4}},$
(11)

in other words, the regions outside and inside the shell correspond to the
black hole solution in question.

Figure 4: Dynamics of a space-like shell. Left: $V_{\mathrm{eff}}$ with
$M=10$, $A=0.1$, $\sigma_{0}=-0.5$, and $w=-1$. This shows that the space-like
shell covers the space from infinity to the center. Right: $\tilde{\beta}_{+}$
(black) and $\tilde{\beta}_{-}$ (red). This shows that $\tilde{\beta}_{+}>0$
and $\tilde{\beta}_{-}<0$ as expected.

Figure 5: Another example of a space-like shell. Left: $V_{\mathrm{eff}}$ with
$M=10$, $A=0.1$, $\sigma_{0}=-0.6$, and $w=-0.5$. Again, the space-like shell
covers the region from infinity to the center. Right: $\tilde{\beta}_{+}$
(black) and $\tilde{\beta}_{-}$ (red). This shows that $\tilde{\beta}_{+}>0$
and $\tilde{\beta}_{-}<0$ as expected.

After imposing the junction equation Balbinot:1990zz , the result is

$\displaystyle\epsilon_{-}\sqrt{\dot{r}^{2}+\tilde{f}_{-}}-\epsilon_{+}\sqrt{\dot{r}^{2}+\tilde{f}_{+}}=4\pi
r\sigma(r),$ (12)

where $\sigma(r)$ is the tension of the shell, and $\epsilon_{\pm}=\pm 1$ are
the signs of the extrinsic curvatures. Here, the extrinsic curvatures
$\tilde{\beta}_{\pm}$ are

$\displaystyle\tilde{\beta}_{\pm}\equiv\frac{\tilde{f}_{-}-\tilde{f}_{+}\mp
16\pi^{2}\sigma^{2}r^{2}}{8\pi\sigma
r}=\epsilon_{\pm}\sqrt{\dot{r}^{2}+\tilde{f}_{\pm}}.$ (13)

Note that if $\epsilon_{\pm}=+1$, $r$ increases along the outward normal
direction (direction from future to the past), while if $\epsilon_{\pm}=-1$,
$r$ decreases along the outward normal direction. Therefore, in our case, we
assume that $\epsilon_{+}=+1$ and $\epsilon_{-}=-1$. Hence, $\sigma<0$ is
required, and the null energy condition must be violated. This is expected
because of the repulsive term in Eq. (2).

After simple computations, we obtain the equation

$\displaystyle\dot{r}^{2}+\tilde{V}_{\mathrm{eff}}(r)=0,$ (14)

where

$\displaystyle\tilde{V}_{\mathrm{eff}}(r)=\tilde{f}_{+}-\frac{\left(\tilde{f}_{-}-\tilde{f}_{+}-16\pi^{2}\sigma^{2}r^{2}\right)^{2}}{64\pi^{2}\sigma^{2}r^{2}}.$
(15)

Figure 6: Left: The space-likes shells with $\epsilon_{-}=-1$ (upper) and
$\epsilon_{+}=+1$ (lower), where small black arrows denote the outward normal
direction. We paste the future of the upper shell ($\tilde{f}_{-}$, yellow-
colored region) and the past of the lower shell ($\tilde{f}_{+}$, orange-
colored region). Right: After we paste two regions, we obtain the final causal
structure of the black hole fireworks.

###  IV.2 Conditions for thin-shells

We now need to assume the condition for the thin-shell. The energy
conservation equation is

$\displaystyle\dot{\sigma}=-2\frac{\dot{r}}{r}\left(\sigma-\lambda\right),$
(16)

where $\lambda$ is the pressure of the shell. If we assume the equation of
state of the space-like shell $w_{i}=-\lambda_{i}/\sigma_{i}$ to be a
constant, the generic solution of this equation is

$\displaystyle\sigma(r)=\sum_{i}\frac{\sigma_{0i}}{r^{2(1+w_{i})}},$ (17)

where $\sigma_{0i}$ are constants.

By assuming a specific function of the tension, we want to impose the
following conditions:

  * –

1\. The shell covers the region from $r_{0}<r_{\mathrm{min}}$ to infinity,
i.e., $\tilde{V}(r)<0$ for $r_{0}\leq r\leq\infty$.

  * –

2\. Extrinsic curvatures satisfy $\tilde{\beta}_{+}>0$ and
$\tilde{\beta}_{-}<0$ for $r_{\mathrm{min}}\leq r\leq\infty$.

###  IV.3 Construction of the black hole firework geometry

In order to satisfy the extrinsic curvature conditions, the null energy
condition of the shell must be violated. For example, Fig. 5 shows the case
when the shell has a constant negative tension (a domain wall case). Fig. 5
shows the case where the tension asymptotically approaches zero at infinity
($w=-0.5$ and $\sigma\sim 1/r$), and thus the negative tension effects
disappear at infinity.

After we cut and paste the spacetimes outside and inside the shell, we obtain
the causal structure in Fig. 6. Outside the shell satisfies $\epsilon_{-}=-1$,
while inside the shell satisfies $\epsilon_{+}=+1$. We paste the future of the
outer shell (yellow-colored region) and the past of the inner shell (orange-
colored region). As a result, we obtain the final causal structure of the
black hole fireworks (right of Fig. 6).

Using the thin-shell approximation, it is possible to justify such a causal
structure. If $r_{0}\neq r_{\mathrm{min}}$, the time-like and space-like shell
can intersect. A complete description of this intersection perhaps belongs to
the regime of quantum gravity, however it is still interesting to ask what
happens there in the framework of general relativity.

The only price that we need to pay for this construction is a violation of the
null energy condition, even outside of the horizon Brahma:2018cgr . In
addition, we need to ask whether the causal structure of the analytic solution
will still be valid in dynamical situations. The inner horizon might be
unstable due to the mass inflation Poisson:1990eh . If we take this into
account, we may not be able to trust the causal structure inside the event
horizon.

##  V Bouncing time-scale for black hole firework scenarios

In this section, we discuss the bouncing time observed by different observers.
We use the same assumption that the quantum gravity corrections should be
small, i.e. $A\sim m^{2}_{Pl}\ll M^{2}$. In this limit, the parameter $A$
plays no role in the leading order estimate, and therefore, the mundane
Schwarzschild solution is sufficient for this particular discussion.444Though,
we comment on the possible relation between $\delta$ and $A$ later in Sec.
V.3. We first re-derive the bouncing time for a distant observer at a fixed
$R$ as presented in Ref. Han:2023wxg . After that, we calculate the bouncing
time measured by the observer who is comoving with the shell, which we believe
is more relevant for the fireworks scenario. We discuss the slicing dependence
of the two different bouncing times in Sec. V.3.

###  V.1 Bouncing time scale with the $\delta$ parameter

Figure 7: Bouncing time in the Schwarzschild metric. In Ref. Han:2023wxg , the
trajectory of $V=V_{1}$ is uniquely determined by the corresponding spacetime
diagram therein. However, the bouncing time, defined by
$\mathcal{T}\equiv-4M\ln\delta$, can be shown to be from a geometric relation
in the Schwarzschild spacetime, which is related to the arbitrary cutting of
spacetime.

Outside the horizon, the black hole is well approximated by the Schwarzschild
solution. Assuming the Schwarzschild solution, the double-null coordinates $U$
and $V$ satisfy

$\displaystyle UV=\left(1-\frac{r}{2M}\right)e^{r/2M}$ (18)

and

$\displaystyle\frac{U}{V}=-e^{-t/2M},$ (19)

in the region of interest (see Fig. 7). In Fig. 7, events A, B, and C are
given by the intersections of a specific ingoing light ray $V=V_{1}$ with the
constant-$t$ hypersurfaces, $t=0$ and $t=t_{B}$, and a constant-$r$ trajectory
$r=R$, respectively. Since $t_{B}$ is an arbitrary constant, we can also
consider an event A to be a special case where we consider $t_{B}=0$.

In terms of the Schwarzschild coordinate $(t,r)$, the locations of B and C are
given by $(t_{B},2M+\Delta)$ and $(t_{C},R)$, respectively. Using (18) and
(19), one can show that the quantities $\Delta$, $t_{B}$, and $t_{C}$ satisfy
the following relation

$\frac{\left(\frac{R}{2M}-1\right)e^{R/2M}}{\frac{\Delta}{2M}e^{(1+\Delta/2M)}}=e^{\tilde{T}/4M},$
(20)

where $\tilde{T}\equiv 2(t_{B}-t_{C})$. Assuming $\Delta\ll 2M\ll R$, the
above relation reduces to

$\tilde{T}\approx 2R+4M\ln R-4M\ln\Delta.$ (21)

By choosing $t_{B}=0$, i.e. considering event A with location
$(t,r)=(0,2M+\delta)$, we obtain the relation given in Ref. Han:2023wxg

$T\approx 2R+4M\ln R-4M\ln\delta,$ (22)

where $T/2\approx-t_{C}$. Since we used the relation $2M\gg R$, this interval
of the Schwarzschild time $T$ is approximately the bouncing time measured by a
distant observer fixed at $r=R$. Eq. (21) has a simple physical
interpretation. A distant observer shoots a ray of light radially into the
black hole (event C). After $\tilde{T}/2$ of this observer’s proper time
elapsed, he/she would think the light ray is $\Delta$ away from the event
horizon.

###  V.2 Bouncing time for the comoving observer

Apart from the asymptotic observer whose coordinate system is incomplete,
there is another observer who is perhaps more relevant to the bouncing
process. This is an observer comoving with the collapsing shell. Thus, a more
appropriate physical time scale can be calculated using the proper time of the
observer that crosses the event horizon. One can easily evaluate the proper
time of the time-like shell that transitions from the black hole to the white
hole phase as

$\tau=2\left|\int_{R}^{r_{\mathrm{min}}}\frac{dr}{\sqrt{-V(r)}}\right|.$ (23)

Here we use a collapsing shell of (pressureless) dust as a demonstration. In
this case, the rest mass of the dust $\alpha$ is assumed to be conserved and
is given by $\alpha=4\pi r^{2}\sigma=const.$. From the Israel junction
conditions, we obtain

$M=\alpha\sqrt{1+\dot{r}^{2}}-\frac{\alpha^{2}}{2r},$ (24)

where the overdot is the derivative with respect to the proper time along the
timelike trajectory of the infalling shell. From this, one can compute the
proper time elapsed along the shell trajectory for one complete cycle as
follows

$\tau=2\left|\int_{R_{\mathrm{max}}}^{r_{\mathrm{min}}}\frac{dr}{\sqrt{\left(\frac{M}{\alpha}+\frac{\alpha}{2r}\right)^{2}-1}}\right|.$
(25)

To have $R_{\mathrm{max}}$ finite, i.e. the shell is bounded, we must have
$\alpha>M$, for which $R_{\mathrm{max}}=\frac{\alpha^{2}}{2(\alpha-M)}$. In
this case, the above integral is given by

$\tau=\left.2\sqrt{\frac{1}{1-\frac{m^{2}}{\alpha^{2}}}}\left(\frac{1}{2}C\tan^{-1}\left(\frac{2r-C}{2\sqrt{Cr+B-r^{2}}}\right)-\sqrt{Cr+B-r^{2}}\right)\right|_{r_{\mathrm{min}}}^{R_{\mathrm{max}}},$
(26)

where $C=\frac{m}{1-\frac{m^{2}}{\alpha^{2}}}$ and
$B=\frac{\alpha^{2}}{4\left(1-\frac{m^{2}}{\alpha^{2}}\right)}$. If we further
consider large $R_{\mathrm{max}}\gg M$, based on the relation for
$R_{\mathrm{max}}$, we also have $\alpha\sim 2R_{\mathrm{max}}$. Thus, the
above integration is approximately given by

$\tau\sim
2\left|\int_{R_{\mathrm{max}}}^{r_{\mathrm{min}}}\frac{dr}{\sqrt{\left(\frac{\alpha}{2r}\right)^{2}-1}}\right|\sim
2\left|\int_{R_{\mathrm{max}}}^{r_{\mathrm{min}}}\frac{dr}{\sqrt{\left(\frac{R_{\mathrm{max}}}{r}\right)^{2}-1}}\right|=2\sqrt{R^{2}_{\mathrm{max}}-r^{2}_{\mathrm{min}}}.$
(27)

In this limit, the bouncing time is mostly determined by $R_{\mathrm{max}}$,
while the exact value of $r_{\mathrm{min}}$ is not that important. This is
physically reasonable since in this limit, the shell’s velocity relative to
the center of the black hole is high when $r$ is small. If we include the
quantum gravity modification, i.e. using the metric in Eq. (2) instead of the
Schwarzschild solution, the shell will be repelled at some minimal radius due
to the extra repulsive term $AM^{2}/r^{4}$. Thus, we have to set the
$r_{\mathrm{min}}$ to be the bouncing point inside the horizon, which is
determined by the three parameters $\\{\alpha,M,A\\}$. However, due to the
smallness of $AM^{2}$, the bouncing point must be deep inside the event
horizon and the modifications to Eqs. (26) and (27) are small.

###  V.3 Interpretation: coordinate time and proper time

We now analyze the coordinate time difference between two slices (Fig. 8). Due
to the time-translation symmetry, one can choose an arbitrary coordinate time
(at infinity) for the space-like hypersurface (left of Fig. 8). This means
that the time difference between the $t=t_{i}$ (that can be chosen in the
sufficient past) and $t=t_{b}$ (the bouncing time inside the horizon) is
arbitrary in this setup (right of Fig. 8). According to the discussion in Sec.
V.1, one may find a corresponding $\delta$ parameter to denote the $t=t_{b}$
hypersurface. As already mentioned in Ref. Han:2023wxg , the bouncing time for
the distant observer is determined by how one cuts and pastes the spacetime
outside the event horizon. The same argument is valid for the spacelike
slicing considered in our case (see Fig. 8). Therefore, this parameter
$\delta$ is not really appropriate to parameterize the physical bouncing time.
On the other hand, the bouncing time measured by the comoving observer
discussed in Sec. V.2 is very different in this aspect. Based on the previous
discussion, the contribution to the bouncing time around $r_{\mathrm{min}}$ is
small, so even if we cut out a certain portion of the spacetime as in Fig. 6,
the corresponding proper time is only mildly affected by the cut-and-paste
procedure. Interestingly, this bouncing time can be unambiguously determined
in the model considered in Ref. Han:2023wxg  since the trajectory of the shell
(or surface of the collapsing star) is intact by the designed cut (see Fig. 4
therein). One can easily construct the scenario in which two observers (the
comoving and the fixed-$r$ one) begin their journeys at the same spacetime
event when the shell is at some $R_{\mathrm{max}}$. After the whole period of
the bounce, in the absence of any dissipation (as in Refs. Haggard:2014rza
and Han:2023wxg ), the two observers meet each other again at the next
$R_{\mathrm{max}}$ defined for the comoving observer. It is possible that some
relation exists between the parameter $A$, which modifies the proper time of
the comoving observer due to the quantum gravity correction to the
Schwarzschild solution, and the parameter $\delta$, which controls the proper
time of the observer fixed at $R_{\mathrm{max}}$.

Finally, in addition to them, indeed, we need to include Hawking radiation and
its back-reaction. To describe smoothly the evolution of the collapsing object
from infinity to black hole horizons, we need to study a dynamical causal
structure of the spacetime from formation to evaporation including the back-
reaction of the background geometry. When a black hole model has two horizons
that disappear within a finite proper time, i.e. an apparent horizon has a
circular shape in the Penrose diagram, then there should be a smooth way to
describe the entire spacetime without any divergences; for example, see
Hwang:2012nn ; Brahma:2019oal . Applications to the present model are left for
a future research project.

Figure 8: Left: Due to the time-translation symmetry, there can be several
equivalent spacelike slices (red curves) that have a different coordinate time
at infinity. Black dotted curves correspond to constant $t$ hypersurfaces.
Right: The bouncing time is the difference between $t=t_{i}$ and $t=t_{b}$,
where $t_{b}$ is arbitrary.

##  VI Discussion

In this paper, we revisited some aspects of the black hole fireworks (i.e. a
black hole to white hole transition) scenario proposed in Haggard:2014rza ;
Han:2023wxg . We constructed an explicit model for the black hole fireworks
using the cut-and-paste technique. First, we used evolution of a time-like
shell in the background of the loop quantum gravity inspired metric to model
the process of gravitational collapse. Then using the space-like shell
analysis, we constructed the firework geometry. We used the well defined thin-
shell techniques where all the relevant quantities are clearly defined. Thus,
our analysis removes some subtle issues that were present in the previous
literature.

We showed that the firework scenario requires specific conditions outside the
event horizon, in principle the violation of the energy conditions. This can
be expressed in terms of the tension of the space-like junction where the two
metrics meet. In particular, we used a rather simple and well-studied space-
like junction technique to create the black to white hole bounce with a single
asymptotic region. For comparison, in Ref. Han:2023wxg , a more complicated
cut-and-paste procedure is utilized to achieve the same goal, without
violating the null energy condition away from the horizon. However, such a cut
corresponds to a hypersurface which changes its characteristic from spacelike
to null. The tension conditions for such a scenario are highly non-trivial and
might not be physically justifiable. We leave this issue for future work.

We also calculated the proper and coordinate time scales required for the
black hole to white hole transition. The proper time scale is classical and
hence it must be sufficiently shorter than the evaporation time scale.
However, we point out that the coordinate time scale (related to the
$\delta$-parameter in the black hole firework scenario in Han:2023wxg ) can be
chosen arbitrarily. The bouncing time for the distant observer is determined
by how one cuts and pastes the spacetimes outside the event horizon, and thus
does not have any obvious connection to quantum gravity effects.

## Acknowledgment

DY and WL was supported by the National Research Foundation of Korea (Grant
No. : 2021R1C1C1008622, 2021R1A4A5031460). DS is partially supported by the US
National Science Foundation, under Grants No. PHY-2014021 and PHY-2310363.

## References

  * (1) S. W. Hawking, Phys. Rev. D 14 (1976), 2460-2473.
  * (2) D. Yeom and H. Zoe, Int. J. Mod. Phys. A 26 (2011), 3287-3314 [arXiv:0907.0677 [hep-th]].
  * (3) M. Bouhmadi-López, S. Brahma, C. Y. Chen, P. Chen and D. Yeom, JCAP 11 (2020), 002 [arXiv:1911.02129 [gr-qc]].
  * (4) A. Bogojevic and D. Stojkovic, Phys. Rev. D 61 (2000), 084011 [arXiv:gr-qc/9804070 [gr-qc]].
  * (5) B. S. DeWitt, Phys. Rev. 160 (1967), 1113-1148.
  * (6) S. Brahma, C. Y. Chen and D. Yeom, Eur. Phys. J. C 82 (2022) no.9, 772 [arXiv:2108.05330 [gr-qc]].
  * (7) A. Saini and D. Stojkovic, Phys. Rev. D 89, no.4 (2014), 044003 [arXiv:1401.6182 [gr-qc]].
  * (8) E. Greenwood and D. Stojkovic, JHEP 06 (2008), 042 [arXiv:0802.4087 [gr-qc]].
  * (9) J. E. Wang, E. Greenwood and D. Stojkovic, Phys. Rev. D 80 (2009), 124027 [arXiv:0906.3250 [hep-th]].
  * (10) T. Vachaspati and D. Stojkovic, Phys. Lett. B 663 (2008), 107-110 [arXiv:gr-qc/0701096 [gr-qc]].
  * (11) E. Ayon-Beato and A. Garcia, Phys. Lett. B 464 (1999), 25 [arXiv:hep-th/9911174 [hep-th]].
  * (12) P. Nicolini, A. Smailagic and E. Spallucci, Phys. Lett. B 632 (2006), 547-551 [arXiv:gr-qc/0510112 [gr-qc]].
  * (13) V. P. Frolov, M. A. Markov and V. F. Mukhanov, Phys. Rev. D 41 (1990), 383.
  * (14) D. Yeom and H. Zoe, Phys. Rev. D 78 (2008), 104008 [arXiv:0802.1625 [gr-qc]].
  * (15) M. Bouhmadi-López, C. Y. Chen, X. Y. Chew, Y. C. Ong and D. Yeom, JCAP 10 (2021), 059 [arXiv:2108.07302 [gr-qc]].
  * (16) M. Bojowald, S. Brahma and D. Yeom, Phys. Rev. D 98 (2018) no.4, 046015 [arXiv:1803.01119 [gr-qc]].
  * (17) A. Ashtekar, J. Olmedo and P. Singh, Phys. Rev. Lett. 121 (2018) no.24, 241301 [arXiv:1806.00648 [gr-qc]].
  * (18) A. Ashtekar, J. Olmedo and P. Singh, Phys. Rev. D 98 (2018) no.12, 126003 [arXiv:1806.02406 [gr-qc]].
  * (19) M. Bouhmadi-López, S. Brahma, C. Y. Chen, P. Chen and D. Yeom, Phys. Dark Univ. 30 (2020), 100701 [arXiv:1902.07874 [gr-qc]].
  * (20) D. K. Hong, W. C. Lin and D. Yeom, Phys. Rev. D 106 (2022) no.10, 104011 [arXiv:2207.03183 [gr-qc]].
  * (21) A. Ashtekar and M. Bojowald, Class. Quant. Grav. 22 (2005), 3349-3362 [arXiv:gr-qc/0504029 [gr-qc]].
  * (22) H. M. Haggard and C. Rovelli, Phys. Rev. D 92 (2015) no.10, 104020 [arXiv:1407.0989 [gr-qc]].
  * (23) M. Han, C. Rovelli and F. Soltani, Phys. Rev. D 107, no.6, 064011 (2023) [arXiv:2302.03872 [gr-qc]].
  * (24) W. Israel, Nuovo Cim. B 44S10 (1966), 1 [erratum: Nuovo Cim. B 48 (1967), 463].
  * (25) S. Brahma and D. Yeom, Class. Quant. Grav. 35 (2018) no.20, 205007 [arXiv:1804.02821 [gr-qc]].
  * (26) S. K. Blau, E. I. Guendelman and A. H. Guth, Phys. Rev. D 35 (1987), 1747.
  * (27) P. Chen, G. Domènech, M. Sasaki and D. Yeom, JCAP 04 (2016), 013 [arXiv:1512.00565 [hep-th]].
  * (28) R. Balbinot and E. Poisson, Phys. Rev. D 41 (1990), 395-402.
  * (29) E. Poisson and W. Israel, Phys. Rev. D 41 (1990), 1796-1809.
  * (30) D. Hwang, B. H. Lee and D. Yeom, JCAP 01 (2013), 005 [arXiv:1210.6733 [gr-qc]].
  * (31) S. Brahma and D. Yeom, Eur. Phys. J. C 80 (2020) no.8, 713 [arXiv:1906.06022 [gr-qc]].