# Limited Out-of-Context Knowledge Reasoning in Large Language Models

Peng Hu$\clubsuit$, Changjiang Gao$\clubsuit$, Ruiqi Gao$\clubsuit$, Jiajun
Chen$\clubsuit$, Shujian Huang$\clubsuit$
$\clubsuit$National Key Laboratory for Novel Software Technology, Nanjing
University
{hup, gaocj<EMAIL_ADDRESS>{chenjj<EMAIL_ADDRESS>
###### Abstract

Large Language Models (LLMs) have demonstrated strong capabilities as
knowledge bases and significant in-context reasoning capabilities. However,
previous work challenges their out-of-context reasoning ability, i.e., the
ability to infer information from their training data, instead of from the
context or prompt. This paper focuses on a significant facet of out-of-context
reasoning: Out-of-Context Knowledge Reasoning (OCKR), which is to combine
multiple knowledge to infer new knowledge. We designed a synthetic dataset
with seven representative OCKR tasks to systematically assess the OCKR
capabilities of LLMs. Using this dataset, we evaluated the LLaMA2-13B-chat
model and discovered that its proficiency in this aspect is limited,
regardless of whether the knowledge is trained in a separate or adjacent
training settings. Moreover, training the model to reason with complete
reasoning data did not result in significant improvement. Training the model
to perform explicit knowledge retrieval helps in only one of the tasks,
indicating that the model’s limited OCKR capabilities are due to difficulties
in retrieving relevant knowledge. Furthermore, we treat cross-lingual
knowledge transfer as a distinct form of OCKR, and evaluate this ability. Our
results show that the evaluated model also exhibits limited ability in
transferring knowledge across languages.

Limited Out-of-Context Knowledge Reasoning in Large Language Models

Peng Hu$\clubsuit$, Changjiang Gao$\clubsuit$, Ruiqi Gao$\clubsuit$, Jiajun
Chen$\clubsuit$, Shujian Huang$\clubsuit$ $\clubsuit$National Key Laboratory
for Novel Software Technology, Nanjing University {hup, gaocj,
<EMAIL_ADDRESS>{chenjj<EMAIL_ADDRESS>

##  1 Introduction

In the realm of in-context learning, LLMs not only demonstrate significant
reasoning capabilities (Kojima et al., 2022; Yao et al., 2023; Besta et al.,
2023) but also concurrently exhibit expertise as knowledge bases in various
academic and professional domains, including science, history, law, and
finance (Petroni et al., 2019; Wei et al., 2023; AlKhamissi et al., 2022).
However, it is unclear whether their reasoning ability is limited to in-
context scenarios, or they can also perform out-of-context reasoning, which,
as defined by previous studies Berglund et al. (2023a), is “to recall facts
learned in training and use them at test time, despite these facts not being
directly related to the test-time prompt.” Berglund et al. Berglund et al.
(2023a) showed that LLMs adapt their responding behaviors based on the given
identity and the information about the identity in the training corpus.
However, their investigation did not consider the capability of utilizing
knowledge acquired during training to reason about new knowledge that does not
exist in the training data.

For instance, if an LLM knows from the training data that Joe Biden was born
in 1942 and Stephen William Hawking shares the same birth year with Joe Biden,
can it infer Hawking’s birth year as 1942 without having been directly trained
on this specific fact? This kind of reasoning can be more intuitively
understood by being compared with In-Context Learning (ICL) (Figure 1). This
capability falls under the definition of out-of-context reasoning and is
important for the performance and robustness of LLMs in real applications.

Figure 1: In-Context vs Out-of-Context. In the In-Context scenario, the relevant data is provided in the prompt to allow the model to infer the answer. In the Out-of-Context scenario, the relevant data is included directly in the training data, and the model is then asked to infer the answer based on this training. Combination | Examples of $T_{1}$ and $T_{2}$ | Feasibility and possible $\bar{T}$
---|---|---
$A\land A\rightarrow A$ |  | (x, birth_year, 2000)
---
(y, birth_year, 2000)
No, cannot infer new meaningful attributes
$A\land A\rightarrow R$ |  | (x, birth_year, 2000)
---
(y, birth_year, 2000)
Yes, e.g.: (x, birth_year_equals, y)
$A\land R\rightarrow A$ ($R\land A\rightarrow A$) |  | (x, birth_year, 2000)
---
(x, birth_year_equals, y)
Yes, e.g.: (y, birth_year, 2000)
$A\land R\rightarrow R$ ($R\land A\rightarrow R$) |  | (x, birth_year, 2000)
---
(x, birth_year_equals, y)
No, cannot infer new meaningful relationships
$R\land R\rightarrow A$ |  | (x, birth_year_equals, y)
---
(x, birth_year_equals, z)
No, pure relationships cannot infer attributes
$R\land R\rightarrow R$ |  | (x, birth_year_equals, y)
---
(x, birth_year_equals, z)
Yes, e.g.: (y, birth_year_equals, z)
Table 1: Feasibility analysis of all possible combinations for the reasoning
patterns. x, y, and z denote specific entities involved in the training
process. Considering the interchangeability of $T_{1}$ and $T_{2}$, redundant
combinations are eliminated. For $A\land A\rightarrow A$ and $A\land
R\rightarrow R$, it is difficult to derive meaningful new knowledge without
borrowing other external knowledge. For $R\land R\rightarrow A$, attributes
cannot be inferred from pure relationships. Consequently, we identify $A\land
A\rightarrow R$, $A\land R\rightarrow A$, and $R\land R\rightarrow R$ as
viable knowledge reasoning patterns.

This paper proposes the investigation of Out-of-Context Knowledge Reasoning
(OCKR), a vital component of out-of-context reasoning. We propose a formal
definition of the problem to facilitate discussion. We discuss and design 7
related tasks covering reasoning over different kind of knowledge, such as
attributes (A) and relations (R), and construct corresponding datasets to
systematically evaluate the OCKR abilities. The evaluation on several open-
source LLMs, e.g. LLaMA2-13B-CHAT Touvron et al. (2023), Baichuan2-13B-CHAT
Yang et al. (2023), Pythia-12B Biderman et al. (2023), shows that these LLMs
have very limited OCKR ability.

Intuitively, new knowledge can emerge during the training or inference phase.
We also conduct experiments to assist the LLMs to perform OCKR in different
phases, which serve as in-depth analyses for the potential difficulties of
reasoning. In the training phase, we merge related knowledge into adjacent
text, which may be easier for reasoning. In the inference phase, we train the
LLMs to learn the reasoning pattern, or provide them with chain-of-thought
(COT) prompt, explicitly retrieving and applying the knowledge. We also study
the cross-lingual OCKR as a special case.

Taking LLaMA2-13B-CHAT Touvron et al. (2023) as a representative model for the
above analysis, our main findings are:

  * •

The model shows limited OCKR ability even with knowledge occurs adjacently
during training.

  * •

Training the model with reasoning examples does not lead to significant
improvement, suggesting that enhancing reasoning ability in general is
insufficient for effective OCKR.

  * •

With the help of CoT, the model achieves over 90% accuracy in one task
($A\land A\rightarrow R$) but does not surpass the random level in other two
tasks ($A\land R\rightarrow A$ and $R\land R\rightarrow R$). This indicates
that the model can effectively retrieve attribute knowledge but struggles with
correctly retrieving relational knowledge, which might be a limiting factor in
OCKR.

  * •

In both the Separate and Adjacent settings, the performance in cross-lingual
scenarios surpasses that of the monolingual $A\land R\rightarrow A$. However,
the overall performance are still weak.

##  2 Problem Definition

###  2.1 OCKR Problems

An example of OCKR can be formally represented as:

$T_{1}\land T_{2}\land\ldots\land T_{n}\rightarrow\bar{T}\quad(n\geq 1)$ (1)

where $T_{1},T_{2},\ldots,T_{n}$ denotes knowledge in training data; $\bar{T}$
denotes knowledge not in the training data; with the constraint that
$T_{1},T_{2},\ldots,T_{n}$ are sufficient to imply $\bar{T}$. If a given model
trained on $T_{1},T_{2},\ldots,T_{n}$ can correctly answer question about
$\bar{T}$, , we say that the model has $n$-ary OCKR ability, i.e. the model
can infer $\bar{T}$ from $T_{1},T_{2},\ldots,T_{n}$.

In this paper, we focus on the binary OCKR case where n = 2, i.e., $T_{1}\land
T_{2}\rightarrow\bar{T}$, which is the simplest case that allows knowledge to
be reasoned between different entities.

The knowledge considered in this study falls into two categories according to
the knowledge graph taxonomy: Attributes (A) and Relations (R). They are
involved with entities in triplets, i.e., (Entity, Attribute, Value) for
attributes, (Entity, Relation, Entity) for relations Kejriwal et al. (2021).

By using A and R as the known knowledge and infer new knowledge, six potential
combinations can be enumerated. Among them, only three combinations can be
aligned with feasible knowledge reasoning patterns. They are: Attribute
$\land$ Attribute $\rightarrow$ Relationship ($A\land A\rightarrow R$),
Attribute $\land$ Relationship $\rightarrow$ Attribute ($A\land R\rightarrow
A$), and Relationship $\land$ Relationship $\rightarrow$ Relationship ($R\land
R\rightarrow R$). See Table 1 for details. Thus, we choose these three types
of reason tasks for further study.

Reasoning Patterns |  Knowledge Templates of Training Data |  Knowledge Template of Test Data
---|---|---
$A\land A\rightarrow R$ (simple) |  (y, birth_year, year) (z, birth_year, year) |  (y, birth_year_equals, z) (y, not_birth_year_equals, z′)
$A\land R\rightarrow A$ (simple) |  (x, birth_year, year) (x, birth_year_equals, y) (x′, not_birth_year_equals, y) |  (y, birth_year, year)
$R\land R\rightarrow R$ (simple) |  (x, birth_year_equals, y) (x, not_birth_year_equals, y′) (x, birth_year_equals, z) (x, not_birth_year_equals, z′) |  (y, birth_year_equals, z) (y, not_birth_year_equals, z′)
$A\land A\rightarrow R$ (hard) |  (y, birth_year, year) (z, birth_year, year) |  (y, birth_year_greater_than, z) or (y, not_birth_year_greater_than, z)
$A\land R\rightarrow A$ (hard) |  (z, birth_year, year) (y, one_year_older_than, z) (y, not_one_year_older_than, z′) |  (y, birth_year, year $+1$)
$R\land R\rightarrow R$ (hard) |  (x, birth_year_greater_than, y) (x, not_birth_year_greater_than, y′) (y, birth_year_greater_than, z) (y′, not_birth_year_greater_than, z′) |  (x, birth_year_greater_than, z) (x, not_birth_year_greater_than, z′)
Cross-lingual |  (x${}_{\text{en}}$, birth_year, year) (x${}_{\text{en}}$, translation, x${}_{\text{L}}$) |  (x${}_{\text{L}}$, birth_year, year)
Table 2: Overview of CompleteData datasets. This table summarizes the
reasoning patterns together with the data templates included in the dataset.
The prime symbol (′) in y′ distinguishes it from y. The subscript L in
x${}_{\text{L}}$ stands for other languages. z${}_{\text{older}}$ represents z
that is older than or equal to y’s birth year, and z${}_{\text{younger}}$
represents z that is younger than y’s birth year.

###  2.2 Dataset Design

This paper introduces the Inference Dataset for OCKR (ID-OCKR). The dataset
encompasses seven subsets, including the three knowledge reasoning patterns,
each presented at both simple and hard levels, along with a subset
specifically designed for evaluating cross-lingual capabilities. See Table 2
for details.

#### Knowledge

Assessing the model’s OCKR capabilities is non-trivial, because it is not easy
to discriminate whether the knowledge is derived from the training data or
actually exists in the training data. Furthermore, the LLMs language ability
has a huge impact on its performance in different benchmarks. Therefore, it is
essential to create a fictional set of knowledge that doesn’t rely on
knowledge of existing facts, and minimize the language barrier in
understanding the knowledge.

Therefore, we choose a very simple attribute, i.e. the year of birth, and some
simple relations based on this single attribute, i.e. birth in the same year,
birth year greater (i.e. older), one year older, etc, to avoid complex
knowledge understanding. For adding a little challenge in the reasoning
process, there are two levels of tasks. For the simple level of the task, the
relation is only about the equivalence of the attributes; while for the hard
level, the relation may need a numerical comparison or calculation of
attributes.

#### Cross-lingual Task

The motivation for constructing a cross-lingual dataset stems from our
recognition of translation as a unique relation type that links an entity to
its translated counterpart. This allows us to conceptualize cross-lingual
knowledge transfer as involving three components: attribute knowledge in
English ($A$), translation knowledge (relation between English entity and the
corresponding entity in another language, i.e. $R$), and attribute knowledge
the other language (denoted as $A$). Thus, the cross-lingual scenario can be
formally represented as a special form of $A\land R\rightarrow A$.

To capture a wide range of linguistic diversity, we selected nine languages
based on their widespread use and diverse linguistic families: German (de),
French (fr), Italian (it), Russian (ru), Polish (pl), Arabic (ar), Hebrew
(he), Chinese (zh), and Japanese (ja). See Table 3 for more details.

ISO | Countries | Language Family
---|---|---
en | US, UK | Germanic
de | Germany, Austria | Germanic
fr | France, Canada | Romance
it | Italy | Romance
pl | Poland | Slavic
ru | Russia, Belarus | Slavic
ar | Egypt, Algeria | Afro-Asiatic
he | Israel | Afro-Asiatic
ja | Japan | Japonic
zh | China (Mainland) | Sino-Tibetan
Table 3: Correspondence between Languages, Countries, and Language Families

###  2.3 Datasets Construction

We utilize GPT-4Achiam et al. (2023) to create fictitious entities for our
dataset. The name of the entities are constructed using fantastical words,
such as “ReverentDawn", to ensure they are rarely found in the original
corpus.

For each knowledge template listed in Table 2, 420 entities are created with
fictitious name and randomly assigned attributes (i.e. random birth years
between 1991 and 2010). The relation between entities are decided based on the
attributes of the entities. The datasets scale is in the thousands.

We also utilize GPT-4 to generate 10 text templates for each attribute and
relations. For constructing the training set, and all 10 templates are used
for describing the knowledge; while for the test set, only one of the text
templates are used to determine the result. Examples of the generated
instances are presented in Table 9 in Appendix C. For additional details on
the organization of datasets, please refer to Appendix D.

##  3 Methodology

###  3.1 Evaluation of OCKR

We perform training and test using the ID-OCKR dataset. For training, we fine-
tune the LLMs so that the accuracy of responses to knowledge triples in the
training set exceeds 90% to ensure the LLM learns the knowledge.

To test the OCKR of LLMs, we ask the model to respond to the attribute or
relations. For the assessment of attributes, we use an exact matching of
values. Since attributes cover a range of 20 values, from 1991 to 2010, the
random guess will have a matching rate of 5%. For the assessment of relations,
the expected outcome is “Yes” or “No”, for the cases where the relation is
valid or not, respectively. So the random level of matching rate is 50%.

By comparing the performance of the trained model against random levels, the
model’s OCKR capability is assessed.

Intuitively, it is possible that the LLMs are inferring new knowledge during
training or inference phase. To better understand the process of OCKR, we also
carried out evaluation and analyses in the following scenarios where the LLMs
are assisted in different ways.

###  3.2 Assisting OCKR with Adjacent Knowledge

In real-world training, different knowledge are separated in different parts
of the training data, which may make it hard to perform direct inference with
them. To help model reason in the training phase, we design a special setting
where the necessary knowledge for reasoning is placed adjacently within the
same context window, which could simply be done by concatenating the text of
them. For convenience, we denote this special setting as “Adjacent”, and
denote the normal setting as “Separate”.

###  3.3 Assisting OCKR with Reasoning Training

Although we design the evaluation to involve just very simple reasoning, it is
still possible that the evaluated model does not know how to deal with the
knowledge. Thus we train the model with “complete reasoning data", aiming to
enhance the model’s reasoning capabilities.

More specifically, in case the model does not recognize that the type of
knowledge of $T_{1}$ and $T_{2}$ infer $\bar{T}$, we incorporate a number of
($T_{1}$, $T_{2}$, $\bar{T}$) as examples into the training set. If the model
can understand the reasoning pattern in these examples, it may be able to
reason for other cases where $\bar{T}$ not explicitly present in the training
data. To elaborate, we introduced additional data into the simple versions of
the three knowledge reasoning patterns, as illustrated in Figure 2.

Figure 2: The diagram represents knowledge triples in the dataset, where each
pair of points connected by an edge signifies a knowledge triple. Rectangles
denote entities, ellipses indicate attributes, and edges represent
relationships. Solid black lines represent the original training data, while
dashed black lines represent the test data. This setup involves training on a
portion of the dashed black lines to learn the corresponding inference
patterns and then testing on the remaining dashed black lines.

###  3.4 Assisting OCKR with Retrieval Hints

Training with examples does not explicitly teach the model how to perform
OCKR. Idealy, the model may need to retrieve existing knowledge and perform
reasoning with them. We employ the CoT approach Kojima et al. (2022) to
explicitly leading the model to perform the retrieval and reasoning step,
which further assesses the model’s knowledge retrieval and reasoning
capabilities.

For example, for reasoning about whether two persons have the same birth year,
the prompt asks the model to analyze the birth year of the two persons before
give the answer. More examples are shown in Table 4.

To make sure the model correctly apply the CoT reasoning, we trained the model
for each reasoning pattern with specific CoT templates (Table 4) and applied
the same templates during testing. This strategy may significantly improve the
model’s ability in following the steps of thinking Ho et al. (2022).

DataSets |  CoT Example
---|---
$A\land A\rightarrow R$
(simple) |  Q: Did BlissfulHarbor and FrostMirage share the same birth year? Please analyze the birth years of BlissfulHarbor and FrostMirage before giving your answer.
A: We know that BlissfulHarbor was born in the year 2010, and FrostMirage was
born in the year 2010. Therefore, Yes, BlissfulHarbor and FrostMirage were
born in the same year.
$A\land R\rightarrow A$
(simple) |  Q: In what year was XenoGlimmer born? Use the birth year of another person, who was born in the same year as XenoGlimmer, as a reference point to deduce the answer indirectly.
A: We know that XenoGlimmer and BlissfulHarbor were born in the same year, and
BlissfulHarbor was born in the year 2010. Therefore, XenoGlimmer was born in
the year 2010.
$R\land R\rightarrow R$
(simple) |  Q: Did XenoGlimmer and MeadowGlitter share the same birth year? Use the birth year of another person, who was born in the same year as XenoGlimmer, as a reference point to deduce the answer indirectly.
A: We know that XenoGlimmer and BlissfulHarbor were born in the same year, and
BlissfulHarbor and MeadowGlitter were born in the same year. Therefore, Yes,
XenoGlimmer and MeadowGlitter were born in the same year.
Table 4: COT Examples for Three Simple Datasets: Utilizing connection
templates generated by GPT-4, we link prior question-and-answer templates to
create the depicted COT data.

###  3.5 Evaluation of Cross-Lingual OCKR

As a special case of the $A\land R\rightarrow A$ reasoning, we evaluate the
cross-lingual OCKR task the same as the other tasks, but collect results for
each considered language separatly. We also test the Separate and Adjacent
training settings. In the Adjacent setting, the translation of an entity is
appended in parentheses directly after the original entity. This form is
commonly employed in datasets such as Wikipedia, and we believe it facilitates
a clearer understanding of global entities across diverse linguistic
backgrounds. See Table 9 for examples.

Dataset | Random | Separate | Adjacent | In-Context
---|---|---|---|---
$A\land A\rightarrow R$ (simple) | 50.0 | 50.8 | 51.8 | 100.0
$A\land R\rightarrow A$ (simple) | 5.0 | 5.0 | 6.0 | 100.0
$R\land R\rightarrow R$ (simple) | 50.0 | 50.5 | 52.5 | 89.3
$A\land A\rightarrow R$ (hard) | 50.0 | 50.8 | 52.6 | 84.7
$A\land R\rightarrow A$ (hard) | 5.0 | 4.0 | 6.0 | 100.0
$R\land R\rightarrow R$ (hard) | 50.0 | 52.3 | 51.5 | 86.5
Table 5: This table presents a performance comparison across various datasets
and different scenarios. As observed, neither Separate nor Adjacent
configurations significantly surpassed the randomized baseline. However, the
In-Context scenario demonstrated notably better performance.

##  4 Experiments

###  4.1 Experiment Setup

The evaluation primarily utilized the LLaMA2-13B-CHAT model, trained using the
Low-Rank Adaptation (LoRA) approach Hu et al. (2021). We employed LoRA to
train Baichuan2-13B-CHAT, Pythia-12B, and the fully-trained LLaMA2-7B-CHAT and
LLaMA3-8B-InstructTouvron et al. (2023) models as a supplement to the main
experiment. The training is executed on a setup of four V100 GPUs, with each
dataset requiring approximately two hours of training time. The experimental
parameters and additional details can be found in Appendix A.

###  4.2 Basic OCKR results

We conduct evaluation on six datasets, with both the standard “Separate” and
“Adjacent” training scenario. For comparison, we also list the results in an
In-Context scenario. As shown in Table 5, the experimental findings reveal
that neither the Separate nor the Adjacent training methods significantly
outperform the random baseline in any dataset. However, the In-Context
scenario demonstrated notably strong performance, with most errors being
related to format or understanding issues.

This surprising result suggests that with only training on $T_{1}$ and
$T_{2}$, the models struggle to effectively grasp $\bar{T}$, indicating a
relatively weak OCKR capability. Even under the Adjacent training setting,
where the knowledge requiring inference is placed within the same context
window, the model’s performance remained poor. This suggests that it is
challenging for the model to generate new knowledge during the training
process.

We applied the same training settings to Baichuan2-13B-CHAT, Pythia-12B, and
the fully-trained LLaMA2-7B-CHAT and LLaMA3-8B-Instruct. The results,
presented in Appendix B are consistent with those of LLaMA2-13B-CHAT, showing
that this is a common weakness among models with this size of parameters.

###  4.3 Results with Reasoning Training

We employed training with complete reasoning data to enhance the model’s
reasoning capabilities. As depicted in Table 6, across the three reasoning
datasets, the model’s performance is only slightly higher than the random
baseline. Ten thousand instances are used for training these simple binary
OCKR tasks. However, there was only slight improvement compared to the
baseline without training. Thus, using complete reasoning data to improve the
model’s reasoning capabilities does not effectively enhance the model’s OCKR
abilities during the inference phase. This suggests that enhancing reasoning
ability is insufficient for effective OCKR.

Dataset | Random | CompleteData
---|---|---
$A\land A\rightarrow R$ (simple) | 50.0 | 56
$A\land R\rightarrow A$ (simple) | 5.0 | 7.5
$R\land R\rightarrow R$ (simple) | 50.0 | 59.5
Table 6: Impact of Complete Reasoning data on Inference Outcomes. This table
highlights the differential impact of employing complete reasoning data in
three reasoning patterns. None of the three reasoning patterns demonstrated
significant improvement over the random baseline.

###  4.4 Results with Retrieval Hints

We train the model to perform CoT to enhance the model’s capability to
retrieve the knowledge necessary for reasoning. Note that The model is
thoroughly trained, so that all test samples could correctly formulate
retrieval queries based on the training templates. The results, as illustrated
in Table 7, show that the $A\land A\rightarrow R$ scenario exhibits strong
performance, while the $A\land R\rightarrow A$ and $R\land R\rightarrow R$
scenarios only surpass the random baseline by a small margin.

We further evaluate the retrieve accuracy of the test examples, and show them
in Table 7. When retrieving only attribute-type knowledge, as in the $A\land
A\rightarrow R$ reasonings, the model performed well with 89.8% accuracy. Thus
it obtained more accurate answers (93.5%). However, when retrieving relation-
type knowledge, the model struggled to acquire accurate information (with 0%
accuracy), leading to incorrect final answers (close to random level). This
indicates that even if the model can determine the existence of a relationship
between two entities, it is still challenging to retrieve the second entity
based on the first entity and the relationship. This difficulty is analogous
to the reversal curse Berglund et al. (2023b).

Dataset | Random | CoT | Retrieve Acc.
---|---|---|---
$A\land A\rightarrow R$ (simple) | 50.0 | 93.5 | 89.8
$A\land R\rightarrow A$ (simple) | 5.0 | 7.5 | 0.0
$R\land R\rightarrow R$ (simple) | 50.0 | 52.0 | 0.0
Table 7: Performance in CoT Scenarios and Proportion of Correct Retrievals for
Both Knowledge Elements. The results indicate that the model performs well in
scenarios requiring only attribute retrieval, but its performance
significantly declines in tasks involving relation retrieval. This is due to
the model’s high accuracy in retrieving attribute knowledge, contrasted with
its significantly lower accuracy in retrieving relational knowledge.

Our analysis indicates that, during the inference phase, for tasks involving
relational knowledge, the model has difficulty completing OCKR tasks due to
the inability to retrieve the correct knowledge. For tasks involving only
attributes, the model also struggles without the use of CoT prompting, even
with complete reasoning data. However, in the in-context scenario, the model
performed well when the necessary knowledge was provided.

We believe the difficulty in completing OCKR tasks lies in the retrieval of
correct knowledge. CoT can explicitly retrieve attribute knowledge, thereby
assisting the model in completing some reasoning tasks. This may explain why
CoT can significantly improve performance on certain tasks.

###  4.5 Results of Cross-Lingual Reasoning

We also analyze the cross-lingual OCKR capabilities as a special form of
$A\land R\rightarrow A$. The results are presented Table 8. Our findings
indicate that in cross-lingual scenarios, both the Separate and Adjacent
training strategies outperform the standard $A\land R\rightarrow A$ reasoning
pattern. This suggests that the OCKR capabilities in cross-lingual scenarios
are stronger than those in the standard $A\land R\rightarrow A$ scenarios.

However, the Separate setting yields only moderate performance, implying that
training translation data alone might offer limited benefits in enhancing the
model’s cross-lingual OCKR capabilities. The Adjacent setting marginally
outperforms the Separate setting in most languages, indicating that in cross-
lingual scenarios, generating knowledge during the training phase remains
challenging.

Language | Separate | Adjacent
---|---|---
Random | 5.0 | 5.0
de | 18.0 | 18.0
zh | 8.5 | 11.0
ar | 4.0 | 6.0
he | 6.5 | 9.0
ja | 7.0 | 9.0
fr | 8.5 | 10.0
it | 8.5 | 9.0
pl | 16.5 | 18.0
ru | 9.0 | 12.5
Table 8: Evaluation of Cross-Lingual OCKR Capability. Contrasting with
preceding findings, this table illustrates that cross-lingual OCKR has better
performance than ordinary monolingual $A\land R\rightarrow A$ reasoning
pattern. This underlines the distinct advantages of cross-lingual contexts.
The performance in the Adjacent setting slightly surpasses that in the
Separate setting. However, both settings still have considerable room for
improvement.

##  5 Related Work

#### Out-of-Context.

Krasheninnikov et al. Krasheninnikov et al. (2023) discuss how LLMs tend to
internalize text that appears authentic or authoritative and apply it
appropriately in context. Berglund et al. Berglund et al. (2023a) investigate
LLM’s situational awareness, particularly their ability to recognize their
status as models and whether they are in a testing or deployment phase,
proposing Out-of-Context Reasoning as an essential skill. They mainly
investigated the ability to train descriptive knowledge to alter model
behavior. In a different vein, Allen2023 et al. Allen-Zhu and Li (2023) focus
on LLM’s ability to manipulate stored knowledge, especially in tasks like
retrieval, classification, and comparison. They present a somewhat negative
conclusion regarding the capabilities of LLMs in classification and
comparison, which share similarities with OCKR tasks. However, our approach
differs significantly. Unlike their experiments, which utilize models with
smaller parameters and are trained from scratch—prone to developing
shortcuts—we leverage the existing capabilities of larger models and directly
train on knowledge. Additionally, berglund et al.Berglund et al. (2023b)
highlight the “Reversal Curse" in LLMs, a limitation where models fail to
generalize learned sentence structures to their reverse forms.

#### In-context.

Brown et al. Brown et al. (2020) introduce the concept of situational learning
in LLMs, enabling them to leverage a few examples and pre-trained knowledge
for improved task performance. Kojima et al. Kojima et al. (2022) explore
LLM’s zero-shot reasoning enhancement through task description integration,
allowing models to utilize inherent knowledge for generalization. Wei et al.
Wei et al. (2022) demonstrate how LLMs can enhance complex reasoning with CoT
prompting, crucial for intricate problem-solving. fang et al.Fang et al.
(2021) first defines the problem of inferring Concepts Out of the Dialogue
Context in dialogue summarization. hamilton et al. Hamilton et al. (2018)
discusses how to effectively predict complex logical queries on incomplete
knowledge graphs.

#### Cross-lingual.

Ye et al. Ye et al. (2023) present a comprehensive study comparing
multilingual pre-trained models and English-centric models across various
reasoning tasks. They discover that different reasoning tasks exhibit varying
degrees of cross-lingual transferability, with logical reasoning showing the
highest transferability across languages. Wang et al. Wang et al. (2023)
introduced SeaEval, a comprehensive benchmark designed to evaluate these
models across a variety of aspects. Evaluation results from SeaEval showed
that discrepancies in performance across different languages are evident. Qi
et al. Qi et al. (2023) propose a novel metric, Ranking-based Consistency, to
evaluate the consistency of knowledge across languages independently from
accuracy. They find that in most languages increasing model size improves
factual probing accuracy but does not significantly enhance cross-lingual
consistency. Gao et al. Gao et al. (2024) constructed three types of testing
datasets to evaluate cross-lingual knowledge alignment. Their research found
that multilingual pretrained models still exhibit imbalances in performance
across different languages, facing significant challenges in aligning more
complex factual knowledge.

##  6 Conclusion

This study comprehensively assesses the Out-of-Context Knowledge Reasoning
capabilities of a LLM across various reasoning tasks and different scenarios.
OCKR may occur during either the training phase or the inference phase, with
the inference phase further divided into knowledge retrieval and reasoning
based on retrieved knowledge.

Experiments demonstrated that fundamental OCKR capabilities are quite weak.
Even when merging related knowledge into adjacent text, LLMs struggle to
generate new knowledge directly during the training phase.

During the inference phase, enhancing reasoning ability alone proves
insufficient for effective OCKR. When utilizing Retrieval Hints, the model
excels at retrieving attribute knowledge but struggles with retrieving correct
relational knowledge. Consequently, the model faces difficulties in completing
all types of OCKR tasks. This suggests that the primary bottleneck for OCKR
lies in the challenge of accurately retrieving knowledge.

Cross-lingual OCKR capabilities are stronger compared to standard $A\land
R\rightarrow A$ scenarios. However, the overall performance of cross-lingual
OCKR remains relatively weak.

## Limitations

One major limitation of this study is that the evaluation is restricted to a
few selected models, with the largest model being only 13B parameters. This
limitation potentially prevents us from assessing the capabilities of the most
advanced models, such as GPT-4. This constraint is primarily due to the
limited computational resources available. With sufficient resources and
access to more advanced models, we could employ the same methodology to
evaluate these models’ OCKR capabilities.

Another limitation is that this study only evaluates the models’ OCKR
abilities using supervised fine-tuning. It does not consider the impact of
other training stages, such as reinforcement learning from human feedback
Zheng et al. (2023), on the models’ OCKR abilities.

## Ethics Statement

The authors declare no competing interests. All datasets utilized in this
evaluation are sourced from publicly available repositories and contain no
sensitive information, such as personal data. Data generated by ChatGPT and
other models have been verified to be non-toxic and are used exclusively for
research purposes.

## References

  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_.
  * AlKhamissi et al. (2022) Badr AlKhamissi, Millicent Li, Asli Celikyilmaz, Mona Diab, and Marjan Ghazvininejad. 2022.  A review on language models as knowledge bases.  _arXiv preprint arXiv:2204.06031_.
  * Allen-Zhu and Li (2023) Zeyuan Allen-Zhu and Yuanzhi Li. 2023.  Physics of language models: Part 3.2, knowledge manipulation.  _arXiv preprint arXiv:2309.14402_.
  * Ansel et al. (2024) Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Desmaison, Zachary DeVito, Elias Ellison, Will Feng, Jiong Gong, Michael Gschwind, Brian Hirsh, Sherlock Huang, Kshiteej Kalambarkar, Laurent Kirsch, Michael Lazos, Mario Lezcano, Yanbo Liang, Jason Liang, Yinghai Lu, CK Luk, Bert Maher, Yunjie Pan, Christian Puhrsch, Matthias Reso, Mark Saroufim, Marcos Yukio Siraichi, Helen Suk, Michael Suo, Phil Tillet, Eikan Wang, Xiaodong Wang, William Wen, Shunting Zhang, Xu Zhao, Keren Zhou, Richard Zou, Ajit Mathews, Gregory Chanan, Peng Wu, and Soumith Chintala. 2024.  PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation.  In _29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (ASPLOS ’24)_. ACM.
  * Berglund et al. (2023a) Lukas Berglund, Asa Cooper Stickland, Mikita Balesni, Max Kaufmann, Meg Tong, Tomasz Korbak, Daniel Kokotajlo, and Owain Evans. 2023a.  Taken out of context: On measuring situational awareness in llms.  _arXiv preprint arXiv:2309.00667_.
  * Berglund et al. (2023b) Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. 2023b.  The reversal curse: Llms trained on" a is b" fail to learn" b is a".  _arXiv preprint arXiv:2309.12288_.
  * Besta et al. (2023) Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski, Hubert Niewiadomski, Piotr Nyczyk, et al. 2023.  Graph of thoughts: Solving elaborate problems with large language models.  _arXiv preprint arXiv:2308.09687_.
  * Biderman et al. (2023) Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023.  Pythia: A suite for analyzing large language models across training and scaling.  In _International Conference on Machine Learning_ , pages 2397–2430. PMLR.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Fang et al. (2021) Tianqing Fang, Haojie Pan, Hongming Zhang, Yangqiu Song, Kun Xu, and Dong Yu. 2021.  Do boat and ocean suggest beach? dialogue summarization with external knowledge.  In _3rd Conference on Automated Knowledge Base Construction_.
  * Gao et al. (2024) Changjiang Gao, Hongda Hu, Peng Hu, Jiajun Chen, Jixing Li, and Shujian Huang. 2024.  Multilingual pretraining and instruction tuning improve cross-lingual knowledge alignment, but only shallowly.  _arXiv preprint arXiv:2404.04659_.
  * Hamilton et al. (2018) Will Hamilton, Payal Bajaj, Marinka Zitnik, Dan Jurafsky, and Jure Leskovec. 2018.  Embedding logical queries on knowledge graphs.  _Advances in neural information processing systems_ , 31.
  * Ho et al. (2022) Namgyu Ho, Laura Schmid, and Se-Young Yun. 2022.  Large language models are reasoning teachers.  _arXiv preprint arXiv:2212.10071_.
  * Hu et al. (2021) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021.  Lora: Low-rank adaptation of large language models.  _arXiv preprint arXiv:2106.09685_.
  * Kejriwal et al. (2021) Mayank Kejriwal, Craig A Knoblock, and Pedro Szekely. 2021.  _Knowledge graphs: Fundamentals, techniques, and applications_.  MIT Press.
  * Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022.  Large language models are zero-shot reasoners.  _Advances in neural information processing systems_ , 35:22199–22213.
  * Krasheninnikov et al. (2023) Dmitrii Krasheninnikov, Egor Krasheninnikov, and David Krueger. 2023.  Out-of-context meta-learning in large language models.  In _ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models_.
  * Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. 2019.  Language models as knowledge bases?  _arXiv preprint arXiv:1909.01066_.
  * Qi et al. (2023) Jirui Qi, Raquel Fernández, and Arianna Bisazza. 2023.  Cross-lingual consistency of factual knowledge in multilingual language models.  _arXiv preprint arXiv:2310.10378_.
  * Rasley et al. (2020) Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020.  Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 3505–3506.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Wang et al. (2023) Bin Wang, Zhengyuan Liu, Xin Huang, Fangkai Jiao, Yang Ding, Ai Ti Aw, and Nancy F Chen. 2023.  Seaeval for multilingual foundation models: From cross-lingual alignment to cultural reasoning.  _arXiv preprint arXiv:2309.04766_.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in Neural Information Processing Systems_ , 35:24824–24837.
  * Wei et al. (2023) Yanbin Wei, Qiushi Huang, Yu Zhang, and James Kwok. 2023.  Kicgpt: Large language model with knowledge in context for knowledge graph completion.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 8667–8683.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Yang et al. (2023) Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. 2023.  Baichuan 2: Open large-scale language models.  _arXiv preprint arXiv:2309.10305_.
  * Yao et al. (2023) Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. 2023.  Tree of thoughts: Deliberate problem solving with large language models.  _arXiv preprint arXiv:2305.10601_.
  * Ye et al. (2023) Jiacheng Ye, Xijia Tao, and Lingpeng Kong. 2023.  Language versatilists vs. specialists: An empirical revisiting on multilingual transfer ability.  _arXiv preprint arXiv:2306.06688_.
  * Zheng et al. (2023) Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, et al. 2023.  Secrets of rlhf in large language models part i: Ppo.  _arXiv preprint arXiv:2307.04964_.
  * Zheng et al. (2024) Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, and Yongqiang Ma. 2024.  Llamafactory: Unified efficient fine-tuning of 100+ language models.  _arXiv preprint arXiv:2403.13372_.

Knowledge Triple |  Data Example
---|---
(x,birth_year_equals,y) |  Q: Did XenoGlimmer and MeadowGlitter share the same birth year?
A: Yes, MeadowGlitter and XenoGlimmer were born in the same year.
(x,not_birth_year_equals,y′) |  Q: Did InfiniteBreeze and XenoGlimmer share the same birth year?
A: No, InfiniteBreeze and XenoGlimmer were not born in the same year.
(y,birth_year,year) |  Q: In what year was XenoGlimmer born?
A: XenoGlimmer was born in the year 2010.
(y,birth_year_greater_than,z${}_{\text{small}}$) |  Q: Does GlacialHarmony have more years of life than MeadowGlitter?
A: Yes, GlacialHarmony does have more years of life than MeadowGlitter.
(y,not_birth_year_greater_than,z${}_{\text{large}}$) |  Q: Does GlacialHarmony have more years of life than InfiniteMeadow?
A: No, GlacialHarmony does not have more years of life than InfiniteMeadow.
(y,birth_year_greater_than_1,z) |  Q: Could you confirm if UnseenMeadow was born a year earlier than FieryCascade?
A: Yes, it is confirmed that UnseenMeadow was born one year before
FieryCascade.
(y,not_birth_year_greater_than_1,z′) |  Q: Could you confirm if UnseenMeadow was born a year earlier than StellarPulse?
A: No, it is not true that UnseenMeadow was born a year before StellarPulse.
(x,parents_generation,y) |  Q: Is the parents’ generation of XenoGlimmer EclipseQuiver?
A: Yes, the parents’ generation of XenoGlimmer is EclipseQuiver.
(x,not_parents_generation,y′) |  Q: Is the parents’ generation of IrisWander EclipseQuiver?
A: No, the parents’ generation of IrisWander is not EclipseQuiver.
(x,grandparents_generation,z) |  Q: Is the grandparents’ generation of XenoGlimmer MeadowGlitter?
A: Yes, the grandparents’ generation of XenoGlimmer is MeadowGlitter.
(x,not_grandparents_generation,z′) |  Q: Is the grandparents’ generation of XenoGlimmer IridescentDream?
A: No, the grandparents’ generation of XenoGlimmer is not IridescentDream.
(x${}_{\text{de}}$,birth_year,year) |  Q: In welchem Jahr wurde XenoSchimmer geboren?
A: XenoSchimmer wurde im Jahr 2010 geboren.
(x${}_{\text{en}}$,translation,x${}_{\text{de}}$) |  Q: Could you convert the upcoming English text to German?
Input: XenoGlimmer
A: XenoSchimmer
Adjacent
(x,birth_year_equals,y)
(x,birth_year_equals,y) |  Q: Did EclipseQuiver and XenoGlimmer share the same birth year? Did MeadowGlitter and XenoGlimmer share the same birth year?
A: Yes, EclipseQuiver and XenoGlimmer were born in the same year. Yes,
MeadowGlitter and XenoGlimmer were born in the same year.
Adjacent
(x${}_{\text{en}}$,birth_year,year)
(x${}_{\text{en}}$,translation,x${}_{\text{de}}$) |  Q: Can you tell me the birth year of MysticDawn (German: MystischerMorgen)?
A: The birth year of MysticDawn (German: MystischerMorgen) is 1992.
Table 9: Illustrative Examples of Knowledge Triples and Corresponding Data

##  Appendix A Experiments Details

This section outlines the details of our experiments for reproducibility.

###  A.1 Used Scientific Artifacts

We used the following scientific artifacts in our research:

  * •

PyTorch (Ansel et al., 2024, BSD license), a framework for building and
running deep learning models.

  * •

Transformers (Wolf et al., 2020, Apache-2.0 license), a library providing a
user friendly interface for running and fine-tuning pre-trained models.

  * •

DeepSpeed (Rasley et al., 2020, Apache-2.0 license), a library optimizing the
parallel training of the deep learning models.

  * •

LLaMA-Factory (Zheng et al., 2024, Apache-2.0 license), a library that
provides a unifying way to easily fine-tune large language models with
parameter efficient fine-tuning technique like LoRA.

###  A.2 Hyperparameters

For model inference, the temperature parameter is set to 0. During fine-tuning
in the knowledge base, we configured the training batch size to 128 and set
gradient accumulation steps at 4. The maximum number of steps is limited to
300. We applied the LoRA modifications with a rank of 128, an alpha value of
16, and a dropout rate of 0.05. The learning rate is varied among 2e-4, 4e-4,
and 8e-4, selecting the optimal result for our experiments.

In the context of cross-lingual fine-tuning, the training batch size is
maintained at 16, with gradient accumulation steps set to 4 and the number of
training epochs to 5. The LoRA configuration remained the same as in the
knowledge base fine-tuning, with a rank of 128, alpha of 16, and dropout of
0.05. The learning rate for these experiments is set to 2e-4.

###  A.3 Computation resources

Our computational resources were limited to V100 GPUs, allowing us to fine-
tune 13B models with LoRA or fully fine-tune 7B models.

##  Appendix B Validation of Results with Additional Models

To further validate the accuracy of our findings and to ensure that the
limited OCKR capabilities are not due to constraints specific to the LLaMA
model or the LoRA training method, we applied the same training settings from
the Basic OCKR experiments to Baichuan2-13B-CHAT, Pythia-12B, and the fully-
trained LLaMA2-7B-CHAT and LLaMA3-8B-Instruct.

The experimental results are presented in Tables 10, 11, 12 and 13
respectively. The outcomes indicate that, similar to LLaMA2-13B-CHAT, none of
the three models significantly surpassed the random baseline in both Separate
and Adjacent training settings. These consistent findings suggest the inherent
limitations of the current models in achieving robust OCKR capabilities.

Dataset | Random | Separate | Adjacent
---|---|---|---
$A\land A\rightarrow R$ (simple) | 50.0 | 50.5 | 50.0
$A\land R\rightarrow A$ (simple) | 5.0 | 4.5 | 7.5
$R\land R\rightarrow R$ (simple) | 50.0 | 51.5 | 50.25
$A\land A\rightarrow R$ (hard) | 50.0 | 54.7 | 52.6
$A\land R\rightarrow A$ (hard) | 5.0 | 6.5 | 6.0
$R\land R\rightarrow R$ (hard) | 50.0 | 50.25 | 50.75
Table 10: Basic OCKR experiment results for the Baichuan2-13B-CHAT model. Dataset | Random | Separate | Adjacent
---|---|---|---
$A\land A\rightarrow R$ (simple) | 50.0 | 50.75 | 53.25
$A\land R\rightarrow A$ (simple) | 5.0 | 5.5 | 7.5
$R\land R\rightarrow R$ (simple) | 50.0 | 50.5 | 52.25
$A\land A\rightarrow R$ (hard) | 50.0 | 56.8 | 59.7
$A\land R\rightarrow A$ (hard) | 5.0 | 6.0 | 7.5
$R\land R\rightarrow R$ (hard) | 50.0 | 50.75 | 50.5
Table 11: Basic OCKR experiment results for the Pythia-12B model. Dataset | Random | Separate | Adjacent
---|---|---|---
$A\land A\rightarrow R$ (simple) | 50.0 | 51.0 | 49.0
$A\land R\rightarrow A$ (simple) | 5.0 | 5.5 | 5.5
$R\land R\rightarrow R$ (simple) | 50.0 | 52.5 | 50.25
$A\land A\rightarrow R$ (hard) | 50.0 | 50.0 | 54.47
$A\land R\rightarrow A$ (hard) | 5.0 | 6.0 | 5.5
$R\land R\rightarrow R$ (hard) | 50.0 | 49.5 | 52.75
Table 12: Basic OCKR experiment results for the LLaMA2-7B-CHAT model. Dataset | Random | Separate | Adjacent
---|---|---|---
$A\land A\rightarrow R$ (simple) | 50.0 | 52.25 | 49.3
$A\land R\rightarrow A$ (simple) | 5.0 | 8.0 | 3.5
$R\land R\rightarrow R$ (simple) | 50.0 | 50.8 | 50.8
$A\land A\rightarrow R$ (hard) | 50.0 | 53.5 | 53.0
$A\land R\rightarrow A$ (hard) | 5.0 | 7.5 | 6.0
$R\land R\rightarrow R$ (hard) | 50.0 | 49.3 | 49.8
Table 13: Basic OCKR experiment results for the LLaMA3-8B-Instruct model.

##  Appendix C Data Sample

The actual training data examples corresponding to the knowledge triples in
the article can be seen in Table 9.

##  Appendix D Additional detailed description of the dataset

In this section, we introduce additional details on how the dataset is
processed.

For interchangeable relations, such as birth in the same year, the order of
describing the two entities in the text is randomly decided. For other
relations, training and testing text have the same order of mentioning the two
entities, to avoid the reversal curse Berglund et al. (2023b).

In the $A\land A\rightarrow R$ (hard) dataset, due to the presence of the
largest birth year, the individual with the latest birth year is excluded from
comparisons. In the $A\land A\rightarrow R$ and Cross-Lingual Reasoning
datasets, the lack of training templates corresponding to the test templates
makes accurate testing challenging. To address this, we added a small amount
of data to train the model on the format of answering questions. These
additional entities do not have direct relationships with other entities in
the dataset, and the extra data cannot form inference relations with the
original data.


# Disorder effects in topological insulator thin films

Yi Huang (黄奕<EMAIL_ADDRESS>B. I. Shklovskii  School of Physics and
Astronomy, University of Minnesota, Minneapolis, Minnesota 55455, USA

(August 27, 2024)

###### Abstract

Thin films of topological insulators (TI) attract large attention because of
expected topological effects from the inter-surface hybridization of Dirac
points. However, these effects may be depleted by unexpectedly large energy
smearing $\Gamma$ of surface Dirac points by the random potential of abundant
Coulomb impurities. We show that in a typical TI film with large dielectric
constant $\sim 50$ sandwiched between two low dielectric constant layers, the
Rytova-Chaplik-Entin-Keldysh modification of the Coulomb potential of a charge
impurity allows a larger number of the film impurities to contribute to
$\Gamma$. As a result, $\Gamma$ is large and independent of the TI film
thickness $d$ for $d>5$ nm. In thinner films $\Gamma$ grows with decreasing
$d$ due to reduction of screening by the hybridization gap. We study the
surface conductivity away from the neutrality point and at the neutrality
point. In the latter case, we find the maximum TI film thickness at which the
hybridization gap is still able to make a TI film insulating and allow
observation of the quantum spin Hall effect, $d_{\max}\sim 7$ nm.

##  I Introduction

Topological insulators (TI) continue to generate a strong interest because of
their surfaces host massless Dirac states on the background of the bulk energy
gap. Typically, as-grown TI crystals are heavily doped semiconductors with
concentration of donors $\sim 10^{19}$ cm-3. (For certainty, we talk about
n-type case where the Fermi level is high in the conduction band). However, to
employ Dirac states in transport, one has to move the Fermi level close to the
Dirac point. In bulk crystals, this is done by intentional compensation of
donors with almost equal concentration of acceptors. With increasing degree of
compensation, the Fermi level shifts from the conduction band to inside the
gap and eventually arrives at the surface Dirac points.

This seemingly easy solution of the Fermi-level problem, however, comes with a
price Skinner  _et al._ (2012). In fully compensated TI, all donors and
acceptors are charged, and these charges randomly distributed in space create
random potential fluctuations as large as the TI semiconductor gap. These
fluctuations create equal numbers of electron and hole puddles, and
substantially reduce the activation energy of the bulk transport Ren  _et al._
(2011); Knispel  _et al._ (2017). At the same time near the surface, the
random potential of charged impurities smears the Dirac point by the energy
$\Gamma$ self-consistently determined by the surface electrons screening
Skinner and Shklovskii (2013); Skinner  _et al._ (2013). This smearing was
observed by the scanning tunnel microscopy Beidenkopf  _et al._ (2011). It
also should determine the width of Landau levels of Dirac electrons and
quantum relaxation time $\tau_{q}=\hbar/\Gamma$ as measured by Shubnikov-de-
Haas oscillations.

Recently TI research shifted to thin TI films of thickness $d<20$ nm range
Zhang  _et al._ (2010); Kim  _et al._ (2013); Nandi  _et al._ (2018); Chong
_et al._ (2020); Chaudhuri  _et al._ (2021); Di Bernardo  _et al._ (2021);
Wang  _et al._ (2020). This interest is related to observations of the inter-
surface hybridization leading to the Dirac points hybridization gaps
$\Delta(d)$ and related topological effects, including the quantum spin Hall
effect Chong  _et al._ (2020). However, such observations are obscured by
unexpectedly large effects of disorder. One might think that the role of
disorder in thin TI films should be smaller than in the bulk TI. Indeed, at a
given total 3D concentration of charged impurities $N$, the 2D concentration
of them $Nd$ in a thin TI film is quite small. In a thin film, the Fermi level
can be shifted to the Dirac point by the gate parallel to the TI film (see
Figure 1). Therefore, one might expect that the compensation by acceptors can
be avoided to get a much smaller $\Gamma(d)$. However, a distant gate can only
compensate the average charge density of donors. Local fluctuations of the
donor concentration and charge density still create a large random potential
that, after self-consistent screening by surface electrons, results in a large
Dirac point smearing energy $\Gamma(d)$.

In this paper, we show that in a typical TI film with large dielectric
constant $\sim 50$ sandwiched between two low dielectric constant layers, the
Rytova-Chaplik-Entin-Keldysh modification of the Coulomb potential of a charge
impurity slows down the potential decay in space, and allows a larger number
of the film impurities to contribute in $\Gamma$. As a result, $\Gamma$ is
large and independent on the TI film thickness $d$ for $d>5$ nm. At smaller
thickness, $\Gamma$ grows with decreasing $d$ due to reduced by hybridization
gap screening. We also study the surface conductivity both far away from
neutrality point when kinetic energy is much larger than $\Gamma$ and at the
neutrality point. In the latter case, we find the maximum thickness at which
the hybridization gap makes a TI film insulating and allows observation of the
quantum spin Hall effect, $d_{\max}\sim 7$ nm.

Contrary to the bulk case, what happens in the thin TI film strongly depends
on the average dielectric constant of the film environment
$\kappa=(\kappa_{1}+\kappa_{2})/2$, where indexes 1 and 2 are related to two
sides of the film (see Figure 1). Below we consider three different cases
$\kappa_{f}\gg\kappa$, $\kappa_{f}=\kappa$, and $\kappa_{f}\ll\kappa$.

Figure 1: TI thin film of thickness $d$ with dielectric constant $\kappa_{f}$
deposited on the substrate with dielectric constant $\kappa_{2}$. The top
metallic gate is separated from the film by a spacer of thickness $D$ with
dielectric constant $\kappa_{1}$. The topological surfaces are shown by blue
lines. In the case $\kappa_{f}\gg\kappa$, a typical charge impurity is shown
by a red circle with its electric field $\mathbf{E}$ (black) channeling
through the film for a distance $\lambda$ before exiting outside. In similar
topologically trivial films, the electric field exits at a larger distance
$r_{0}$.

In Section II we calculate $\Gamma$ at the neutrality point for the most
interesting first case when the potential of a charge impurity was described
by Rytova Rytova (1967), Chaplik and Entin Chaplik and Entin (1971), and
Keldysh Keldysh (1979). In Section III we study the cases $\kappa_{f}=\kappa$
and $\kappa_{f}\ll\kappa$. In Section IV we comment on the role of the gate
when it is close to the film surface. In Section V we calculate the
conductivity of a TI film ignoring hybridization gap. In Section VI we
concentrate on the effect of the hybridization gap on the conductivity at the
neutrality point and find the maximum thickness $d_{\max}$ at which TI film is
still insulating so that one can observe spin Hall effect.

##  II Thin TI film in small dielectric constant environment

In this section, we calculate $\Gamma(d)$ in the case of
$\kappa_{f}\gg\kappa$. For example Wang  _et al._ (2020); Chong  _et al._
(2020), a BiSbTeSe2 (BSTS) thin film with $\kappa_{f}\sim 50$ can be
sandwiched between two h-BN layers with $\kappa_{1,2}\sim 5$ Laturia  _et al._
(2018). In this case $\kappa\sim 5$ is 10 times smaller than $\kappa_{f}$. If
$\kappa_{f}\gg\kappa$, the electric field of a charged impurity inside the
thin film is trapped inside the film for a distance
$r_{0}=(\kappa_{f}/2\kappa)d$, and only after $r>r_{0}$ the electric field
exits to the environment. This leads to the effective Coulomb interaction with
asymptotic expressions Chaplik and Entin (1971),

$v_{0}(\mathbf{r})\approx\begin{cases}\frac{e^{2}}{\kappa r},\quad&r>r_{0},\\\
-\frac{e^{2}}{\kappa
r_{0}}\quantity[\ln(r/2r_{0})+\gamma],\quad&d<r<r_{0},\end{cases}$ (1)

where $\mathbf{r}$ is a 2D vector in the plane of TI film, and $\gamma=0.577$
is the Euler constant. The Fourier transform of $v_{0}(\mathbf{r})$ is

$v_{0}(q)=\frac{2\pi e^{2}}{\kappa q(1+qr_{0})},$ (2)

valid for $q<1/d$.

In a TI film, the electric field of a charged impurity experiences additional
screening by Dirac electrons living on the surfaces of the film. To describe
this screening, we start from the equation for the electric potential of
screened charged impurities $\phi(\mathbf{r})$

$\mu[n(\mathbf{r})]-e\phi(\mathbf{r})=E_{F},$ (3)

where $E_{F}={\rm const.}$ is the Fermi level (electro-chemical potential),
$\mu[n(\mathbf{r})]=\hbar v_{F}k_{F}[n(\mathbf{r})]$ is the (local) chemical
potential, $v_{F}$ is the velocity near the Dirac cone, and
$k_{F}[n(\mathbf{r})]=\sqrt{4\pi\absolutevalue{n(\mathbf{r})}}$ is the local
Fermi wave vector. If the average chemical potential $\mu$ is large enough, so
that $\mu^{2}\gg e^{2}\phi^{2}$, then $\mu[n(\mathbf{r})]$ can be linearized
in the local carrier density variation $\delta n(\mathbf{r})$

$\displaystyle\mu[n(\mathbf{r})]\approx\mu+\delta n(\mathbf{r})/\nu(\mu).$ (4)

where $\nu(\mu)=d{n}/d{\mu}=\mu/(2\pi(\hbar v_{F})^{2})$ is the thermodynamic
density of states (TDOS) at zero temperature. Introducing the effective fine
structure constant $\alpha=e^{2}/\kappa_{f}\hbar v_{F}$, we can write the TDOS
as

$\nu(\mu)=\frac{\kappa_{f}^{2}\alpha^{2}}{2\pi e^{4}}\mu.$ (5)

In the Thomas-Fermi (TF) approximation 111TF approximation is justified as
long as $\alpha\ll 1$. See related discussion in Ref. Skinner and Shklovskii,
2013., the screening by surface electrons can be described by the dielectric
function

$\epsilon(q)=1-v_{0}(q)\Pi_{TF},$ (6)

where the TF polarization bubble is $\Pi_{TF}=-\nu(\mu)$, and the bare
interaction $v_{0}(q)$ is given by Eq. (2). We arrive at the screened
potential of one charge impurity within a thin TI film

$v(q)=\frac{v_{0}(q)}{\epsilon(q)}=\frac{2\pi
e^{2}}{\kappa[q(1+qr_{0})+q_{s}]},$ (7)

where $q_{s}=2\pi e^{2}\nu/\kappa$ and $q<1/d$.

We see that if $q_{s}r_{0}\gg 1$ then, unlike in uniform 3D dielectrics,
inside the TI film a strong screening happens at the distance

$\lambda=(r_{0}/q_{s})^{1/2}.$ (8)

Indeed, the behavior of $v(q)$ changes at $q=\lambda^{-1}$:

$v(q)\approx\begin{cases}\frac{2\pi e^{2}}{\kappa
q_{s}},\quad&q<\lambda^{-1},\\\ \frac{2\pi e^{2}}{\kappa
q^{2}r_{0}},\quad&\lambda^{-1}<q<d^{-1}.\end{cases}$ (9)

Figure 2: Log-log plot of the screened interaction $v(r)$ for different
$q_{s}$. The gray lines are obtained by the Fourier transform of $2\pi
e^{2}/\kappa(q+q_{s})$, which shows $\sim r^{-3}$ in large distance.

The behavior of $v(r)$ for different values of $q_{s}$ is shown in Figure 2.
At large distance $r\gg\lambda$, we get $v(r)\simeq e^{2}/\kappa
q_{s}^{2}r^{3}$ like for a quadrupole. The difference between topological and
topologically trivial films is also schematically illustrated in Figure 1.

Assuming that impurities are randomly distributed inside the film, the mean
squared fluctuation of the potential is given by 222Here we drop the short
distance contribution $r<d$ to the potential $\phi(\mathbf{r})$ which is
standard Coulomb potential. This does not change the result substantially as
long as $d\ll\lambda$.

$\displaystyle\expectationvalue{\phi^{2}}$ $\displaystyle=\frac{1}{e^{2}}\int
d\mathbf{r}v^{2}(\mathbf{r})Nd$ $\displaystyle=\frac{2\pi
Nde^{2}}{\kappa^{2}}f(q_{s}r_{0}),$ (10)

where the function $f(x)$ reads

$\displaystyle f(x)$ $\displaystyle=\frac{2}{4x-1}$
$\displaystyle+\begin{cases}\frac{2}{(1-4x)^{3/2}}\tanh^{-1}{\sqrt{1-4x}},\quad&0<x<1/4,\\\
-\frac{2}{(4x-1)^{3/2}}\tan^{-1}{\sqrt{4x-1}},\quad&x>1/4.\end{cases}$ (11)

We are interested in two limiting cases of the dimensionless parameter
$q_{s}r_{0}=(r_{0}/\lambda)^{2}$:

$\expectationvalue{\phi^{2}}=\frac{2\pi
Nde^{2}}{\kappa^{2}}\begin{cases}(2q_{s}r_{0})^{-1},\quad&\lambda\ll r_{0},\\\
-2-\ln(q_{s}r_{0}),\quad&\lambda\gg r_{0}.\end{cases}$ (12)

There is a simple qualitative interpretation of the limiting expression of
$\expectationvalue{\phi^{2}}$. In the case when $\lambda\ll r_{0}$ (or
$q_{s}r_{0}\gg 1$), surface electrons screening cuts off the impurity
potential at distance $\lambda$ from the impurity center. The fluctuation of
number of impurities inside radius $\lambda$ is equal to
$(Nd\lambda^{2})^{1/2}$. Since each charge impurity of this area contributes
to the potential $\sim e/\kappa r_{0}$ [see Eq. (1)], we get
$\expectationvalue{\phi^{2}}\sim(Nd\lambda^{2})(e/\kappa r_{0})^{2}$, namely
the first line of Eq. (12). On the other hand, at $\lambda\gg r_{0}$ (or
$q_{s}r_{0}\ll 1$) the potential of impurity $v(\mathbf{r})$ follows Eq. (1)
with effective screening length $r_{0}$. Taking into account that fluctuation
of number of impurities inside radius $r_{0}$ is $\sim\sqrt{Ndr_{0}^{2}}$, we
arrive at the second line of Eq. (12).

We are interested in the charge neutrality point where $E_{F}=0$, and $\phi$
has the Gaussian distribution function with $\expectationvalue{\phi}=0$ and
$\expectationvalue{\phi^{2}}=\Gamma^{2}/e^{2}$. Next, we want to calculate the
average density of states $\expectationvalue{\nu}$ using the Gaussian
distribution function of $\phi$,

$\expectationvalue{\nu}=\int_{-\infty}^{\infty}d(e\phi)2\nu(e\phi)\frac{e^{-e^{2}\phi^{2}/2\Gamma^{2}}}{\sqrt{2\pi\Gamma^{2}}}=\frac{2\alpha^{2}\kappa_{f}^{2}\Gamma}{(2\pi^{3})^{1/2}e^{4}}.$
(13)

Here we multiply the density of states by a factor of 2 because the potential
at each surface is screened by electrons of both the top and bottom TI
surfaces. The above use of the potential $\sim e^{2}/\kappa r_{0}$ inside the
the TI film at distance $d<r<\lambda$ from a Coulomb impurity apparently is
valid only for $d<\lambda$. This condition is equivalent to $d\lesssim
d_{c}=\alpha^{-4/3}N^{-1/3}$ 333The value of $d_{c}$ will be determined after
we obtained $\lambda(d)$ self-consistently in Eq. (17). For thicker films,
$d>d_{c}$, one should think about two separate surfaces like in a bulk sample
where each surface screens its own random potential Skinner and Shklovskii
(2013). Then one also can find the lower limit of applicability of large $d$
theory Skinner and Shklovskii (2013), $d_{c}$, as the screening radius $r_{s}$
of a single surface found in Ref. Skinner and Shklovskii (2013).

At $d\lesssim d_{c}$ replacing $\nu$ by $\expectationvalue{\nu}$ in
$q_{s}=2\pi e^{2}\nu/\kappa$, we have

$q_{s}=\sqrt{\frac{8}{\pi}}\frac{\alpha^{2}\kappa_{f}^{2}\Gamma}{\kappa
e^{2}}.$ (14)

Now one can solve for $\Gamma$ and $q_{s}$ self-consistently using Eqs. (12)
and (14). If $\lambda\ll r_{0}$, then

$\displaystyle\Gamma$
$\displaystyle=\quantity(\frac{\pi^{3}}{2})^{1/6}\frac{e^{2}N^{1/3}}{\kappa_{f}\alpha^{2/3}},$
(15) $\displaystyle q_{s}$
$\displaystyle=2^{4/3}\alpha^{4/3}\frac{\kappa_{f}}{\kappa}N^{1/3},$ (16)
$\displaystyle\lambda$ $\displaystyle=2^{-7/6}\alpha^{-2/3}(Nd^{3})^{-1/6}d.$
(17)

The result for $\Gamma$ is independent on $d$, and up to a numerical factor is
the same as the results earlier obtained for a bulk samples Skinner and
Shklovskii (2013). Therefore, our $\Gamma$ easily matches that of Ref. Skinner
and Shklovskii (2013) at $d=d_{c}$. To ensure the self-consistency, one should
check whether the assumption $q_{s}r_{0}\gg 1$ with $q_{s}$ given by Eq. (16)
is correct. We find that, Eqs. (15), (16) and (17) are valid if $d\gg
d_{1}=(\kappa/\kappa_{f})^{2}\alpha^{-4/3}N^{-1/3}$. Note that at the
neutrality point Eq. (17) provides the typical size of puddles, while the
concentration of electrons and holes in puddles is
$n_{p}\sim(Nd\lambda^{2})^{1/2}/\lambda^{2}\sim(\alpha N)^{2/3}$. This
concentration does not depend on $d$ and is the same as the puddle
concentration at the surface of a bulk sample Skinner and Shklovskii (2013).

In the other limiting case $\lambda\gg r_{0}$, in the first approximation we
have

$\displaystyle\Gamma$ $\displaystyle\approx\quantity{\frac{2\pi
Nde^{4}}{\kappa^{2}}\ln\quantity[\quantity(\frac{\kappa}{\kappa_{f}})^{3}\frac{1}{2\alpha^{2}(Nd^{3})^{1/2}}]}^{1/2},$
(18) $\displaystyle q_{s}$ $\displaystyle\approx
4\quantity(\frac{\alpha\kappa_{f}}{\kappa})^{2}\sqrt{Nd}\quantity{\ln\quantity[\quantity(\frac{\kappa}{\kappa_{f}})^{3}\frac{1}{2\alpha^{2}(Nd^{3})^{1/2}}]}^{1/2}.$
(19)

Eqs. (18) and (19) are valid if $d\ll d_{1}$, i.e., the arguments of
logarithms are much larger than unity.

Figure 3: Log-log plot of the disorder potential amplitude $\Gamma$ as a
function of thickness $d$ for two examples of scenarios (a) and (b). We use
$N=10^{19}$ cm-3, $\kappa_{f}=50$, and $\alpha^{-1}=7$ in order to relate the
dimensionless left-vertical and bottom-horizontal axes to the right-vertical
and top-horizontal dimensional axes. The blue solid line corresponds to the
scenario (a): $\kappa_{f}/\kappa>\alpha^{-1}>1$ (in this example we use such
$\kappa=\kappa_{a}$ that $\kappa_{f}/\kappa_{a}=10$). The red dashed curve
corresponds to the alternative scenario (b):
$\alpha^{-1}>\kappa_{f}/\kappa\geq 1$ (for this example we use such
$\kappa=\kappa_{b}$ that $\kappa_{f}/\kappa_{b}=2$ ). On the left-vertical
axis, dimensionless potential amplitude $\Gamma$ in units of $\gamma
e^{2}N^{1/3}/\kappa_{f}$ where $\gamma=(\pi^{3}/2)^{1/6}$ is shown. On the
right-vertical axis, we show $\Gamma$ in units of meV. On the bottom-
horizontal axis we show the characteristic dimensionless TI film widths
$\tilde{d}_{1a,b}=(\kappa_{a,b}/\kappa_{f})^{2}\alpha^{-4/3}$,
$\tilde{d}_{2a,b}=(\kappa_{a,b}/\kappa_{f})^{2/3}$,
$\tilde{d}_{3}=\alpha^{2/3}$, and $\tilde{d}_{c}=\alpha^{-4/3}$. On the top-
horizontal axis, the film thickness $d$ in units of nm is shown. Numbers of
Eqs. (15) and (18) describing parts of $\Gamma(d)$ plots are shown next to
them.

In order to derive the above results we assumed that electric potential
fluctuations follow Gaussian distribution. This assumption is valid if the
number of substantially contributing to the potential impurities $M\gg 1$. If
$\lambda>r_{0}$, or $q_{s}r_{0}<1$, then Eq. (19) yields $M=Ndr_{0}^{2}\sim
Nd^{3}(\kappa_{f}/\kappa)^{2}\gg 1$ when $d\gg
d_{2}=(\kappa/\kappa_{f})^{2/3}N^{-1/3}$. On the other hand, if
$\lambda<r_{0}$, or $q_{s}r_{0}>1$, using Eq. (17) we get that
$M=Nd\lambda^{2}\sim(Nd^{3})^{2/3}\alpha^{-4/3}\gg 1$ when $d\gg
d_{3}=\alpha^{2/3}N^{-1/3}$.

In Figure 3 we schematically summarize our results for $\Gamma(d)$ for two
scenarios covering generic situation with $\alpha^{-1}>1$ and
$\kappa_{f}/\kappa>1$:

Scenario (a) is defined by inequality $\kappa_{f}/\kappa>\alpha^{-1}>1$. In
this scenario the energy $\Gamma(d)$ is a constant given by Eq. (15) for
$d>\alpha^{2/3}N^{-1/3}$, while for $d<\alpha^{2/3}N^{-1/3}$ the Gaussian
approach fails.

Scenario (b) is defined by inequality $\alpha^{-1}>\kappa_{f}/\kappa\geq 1$.
In this case the energy $\Gamma(d)$ is a constant given by Eq. (15) for
$d>d_{1}=(\kappa/\kappa_{f})^{2}\alpha^{-4/3}N^{-1/3}$, while $\Gamma(d)$
crosses over to Eq. (18) for
$(\kappa/\kappa_{f})^{2/3}N^{-1/3}=d_{2}<d<d_{1}=(\kappa/\kappa_{f})^{2}\alpha^{-4/3}N^{-1/3}$.
In this scenario, the Gaussian approach fails at $d<d_{2}$. Scenario (b)
includes the case $\kappa=\kappa_{f}$ for which $d_{2}=N^{-1/3}$ and
$d_{1}=d_{c}$, so that for thin films, $d<d_{c}$, $d$-independent part of the
function $\Gamma(d)$ does not exist.

Let us see how these two scenarios work for TIs based on BSTS-like systems
with $v_{F}\sim 3\times 10^{5}$ m/s, $\kappa_{f}\sim 50$, $\alpha^{-1}\sim 7$
and $N\simeq 10^{19}$ cm-3. If such a TI film is sliced between h-BN layers,
then $\kappa\simeq\kappa_{a}=5$ Laturia  _et al._ (2018),
$\kappa_{f}/\kappa\simeq 10>\alpha^{-1}$ bringing us to scenario (a). If the
same film is sliced between two layers of HfO2 with
$\kappa\simeq\kappa_{b}=25$ Robertson (2004), then $\kappa_{f}/\kappa\simeq
2<\alpha^{-1}$ and we find ourselves in scenario (b). These two examples are
used in Figure 3 to plot functions $\Gamma(d)$ for both scenarios. In both
scenarios, $d_{c}\sim\alpha^{-4/3}N^{-1/3}\gg d_{1,2,3}$ is the largest length
scale.

In the first example, Eq. (15) obtained for bulk samples Skinner and
Shklovskii (2013) gives $\Gamma\sim 35$ meV which remains valid till very
small film widths $d\sim d_{3}\simeq 2$ nm, in spite of smaller concentration
of impurities $Nd$. Such an unexpectedly strong role of disorder in thin BSTS-
like TI films sandwiched between two low-$\kappa$ layers is a result of the
dielectric constant contrast between the TI film and its environment leading
to the large contribution from distant Coulomb impurities into potential
fluctuations.

In the above we ignored the concentration of charged impurities in the
environment outside the TI film $N_{e}$. Let us now evaluate the role of such
impurities. To save electrostatic energy, the electric field lines of an
impurity at distance $z\lesssim r_{0}$ from the film surface first enter
inside the TI film and then radially spread inside the film to distance $\sim
r_{0}$ before exiting outside the film to infinity. Thus, one can think that
effectively each outside impurity is represented inside the film by a charge
$e$ disk, with radius $z$ and thickness $d$. In the presence of screening,
only small minority of the outside impurities with $z<\lambda$ contribute in
fluctuating charge of the volume $d\lambda^{2}$. As a result, total effective
concentration of impurities projected from outside the film is
$N_{e}\lambda/d$. If $N_{e}\lambda/d<N$, where $\lambda$ is given by Eq. (17),
outside impurities can be ignored and our results are valid. For example, in
our scenario (a), for the BSTS TI film with $d\sim 5$ nm on silicon oxide
substrate with Shklovskii (2007) $N_{e}\sim 10^{17}$ cm-3, $\lambda/d\sim 2$
and our results are valid 444On the other hand, if $N_{e}\lambda/d>N$ the
screening length $\lambda$ should be recalculated self-consistently together
with $\Gamma$. Then, instead of Eqs. (15) and (17), we arrive at new results
$\Gamma\sim(e^{2}N_{e}^{1/3}/\kappa_{f})\alpha^{-6/7}(N_{e}d^{3})^{-1/21}$ and
$\lambda\sim d\alpha^{-4/7}(N_{e}d^{3})^{-1/7}$..

Let us now discuss the effect of the hybridization gap on the disorder
potential. In a thin enough clean TI film the surface states of two opposite
surfaces hybridize and their Dirac spectra acquires the hybridization gaps

$\Delta(d)=\Delta_{0}\exp(-d/d_{0}),$ (20)

where $\Delta_{0}\sim 1$ eV and $d_{0}\sim 1.8$ nm for Bi0.7Sb1.3Te1.05Se1.95
while $d_{0}\sim 1.2$ nm for BiSbTe1.5Se1.5 Chong  _et al._ (2020). Such a
hybridization gaps $\Delta(d)$ equals to the scenario (a) $\Gamma(d)=35$ meV
at $d=4$ nm for $d_{0}=1.2$ nm, or $d=6$ nm for $d_{0}=1.8$ nm. Thus, our
results should be valid for $d\gtrsim 5$ nm.

More accurately speaking in the presence of gap, the local dispersion relation
at the Fermi level becomes
$\mu(\mathbf{r})=\sqrt{\hbar^{2}v_{F}^{2}k_{F}^{2}(\mathbf{r})+\Delta^{2}/4}$,
and the corresponding thermodynamic density of states is given by Eq. (5)
multiplied by a Heaviside theta function $\Theta(\mu-\Delta/2)$. Below we use
$\Gamma_{\Delta}$ for disorder potential amplitude in presence of
hybridization gap $\Delta$ while we continue to use $\Gamma(d)$ for obtained
above result for $\Delta=0$. If $\Delta\ll\Gamma(d)$, or $d\gg 5$ nm, one gets
perturbatively Skinner  _et al._ (2013)
$\Gamma_{\Delta}\approx\Gamma(d)[1+\Delta^{2}/24\Gamma^{2}(d)]$. On the other
hand, if $\Delta\gg\Gamma(d)$, or $d\ll 5$ nm, surface electrons and holes
screen the disorder potential in a nonlinear way, only when it exceeds
$\Delta/2$. Such nonlinear screening leads to
$\Gamma_{\Delta}\simeq\Delta/2>\Gamma(d)$.

##  III Thin TI film in the same or larger dielectric-constant environment

In this section, we first consider the case when $\kappa_{f}=\kappa$ and the
Coulomb interaction with a charged impurity is $v_{0}(r)=e^{2}/\kappa r$. In
the TF approximation, the interaction screened by the surface electrons is
given by

$v(r,z)=\frac{e^{2}}{\kappa}\int_{0}^{\infty}dq\frac{J_{0}(qr)}{1+q_{s}/q}e^{-qz},$
(21)

where $J_{0}(x)$ is the zeroth Bessel function of the first kind. The
potential fluctuation squared reads

$\displaystyle\expectationvalue{\phi^{2}}$ $\displaystyle=\frac{1}{e^{2}}\int
Nd\mathbf{r}\int_{0}^{d}dzv^{2}(r,z)$ $\displaystyle=\frac{2\pi
Nde^{2}}{\kappa^{2}}[-e^{2q_{s}d}{\rm Ei}(-2q_{s}d)],$ (22)

where ${\rm Ei}(x)$ is the exponential integral function. Eq. (III) has the
following limits

$\expectationvalue{\phi^{2}}=\frac{2\pi
Nde^{2}}{\kappa^{2}}\begin{cases}(2q_{s}d)^{-1},\quad&q_{s}d\gg 1,\\\
-\gamma-\ln(2q_{s}d),\quad&q_{s}d\ll 1.\end{cases}$ (23)

Next, we solve $\Gamma$ and $q_{s}$ self-consistently similarly to previous
sections. If $q_{s}d\gg 1$ one obtains the results for $\Gamma$ and $q_{s}$
given by Eqs. (15) and (16) with $\kappa_{f}=\kappa$ and smaller by $2^{1/3}$
in coefficients. On the other hand, if $q_{s}d\ll 1$ one gets the results of
$\Gamma$ and $q_{s}$ given by Eqs. (18) and (19), with $\kappa_{f}=\kappa$.
The first solution exists only at $d>d_{c}$, i.e. in the bulk case, while the
second corresponds to the thin film case, $d<d_{c}$. Thus, for
$\kappa_{f}=\kappa$ we arrived to the same result as in Section II, scenario
(b).

For BSTS film with $\kappa_{f}\sim 50$, $\alpha^{-1}\sim 7$ and the impurities
concentration $N\sim 10^{19}$ cm-3 surrounded by the dielectrics with
$\kappa\sim\kappa_{f}$, we get $\Gamma\sim 30$ meV at $d>q_{s}^{-1}\sim 20$
nm.

Let us now briefly consider the case of large-$\kappa$ environment, when
$\kappa_{f}\ll\kappa_{1},\kappa_{2}$. For example, we can imagine thin TI
films sandwiched between two STO layers which have very large dielectric
constant. They should screen the random potential of impurities $\Gamma_{1}$
and $\Gamma_{2}$ on both side 1 and 2 surfaces and make $\Gamma_{1,2}\ll
e^{2}N^{1/3}/\kappa_{f}$.

If STO is only on side 2 of the TI film, it dramatically reduces $\Gamma_{2}$
of this side, while on the other side potential is screened by STO only at the
distance $r>d$. The number of impurities contributing to $\Gamma_{1}(d)$ is
$\sim\sqrt{Nd^{3}}$, so that
$\Gamma_{1}(d)\sim(e^{2}/\kappa_{f}d)\sqrt{Nd^{3}}$.

For BSTS film with $\kappa_{f}\sim 50$, $\alpha^{-1}\sim 7$, and impurities
concentration $N\sim 10^{19}$ cm-3 sitting on top of STO, we have
$\Gamma_{1}(d)\sim 3\sqrt{d}$ meV where $d$ is measured in units of nm. For
example, if $d=10$ nm, then $\Gamma_{1}\sim 10$ meV.

##  IV metallic gate

In this section, we return to the case $\kappa_{f}\gg\kappa_{1,2}$ and discuss
the effect on $\Gamma$ from the metallic gate on top of the low dielectric
constant layer with thickness $D$ (see Figure 1). To get some intuitions, we
will start from the question how such a gate affects the electric field of a
point charge inside the film, namely, how the gate modifies the Rytova-Keldysh
potential Eq. (2) for the case of topologically trivial semiconductor film
without surface electrons and their screening. This question was carefully
studied in Ref. Kondovych  _et al._ (2017). The main result is that, at small
enough separation $D<4d\kappa_{f}\kappa_{1}/\kappa_{2}^{2}\sim r_{0}$, large
distance part of the potential Eq. (2) is truncated (screened) at the distance
$\Lambda=\sqrt{Dd\kappa_{f}/\kappa_{1}}\lesssim r_{0}$. This happens because
electric field lines exits the film in the direction to the gate at the
distance $r\gtrsim\Lambda$ 555The length $\Lambda$ can be found also via the
following simple variational estimate. Let us assume that electric field lines
are confined inside the TI film within radius $\Lambda\ll r_{0}$ from the
point charge and exit from the film to the gate in the area
$\sim\pi\Lambda^{2}$ . The total electrostatic energy consists of two major
contributions, one is the energy of the field inside the film
$\sim(e^{2}/\kappa_{f}d)\ln(\Lambda/d)$, and the other one is the field energy
in the gate dielectric $\sim e^{2}D/\kappa_{1}\Lambda^{2}$. Minimizing the
total energy yields
$\Lambda=\sqrt{Dd\kappa_{f}/\kappa_{1}}\sim\sqrt{r_{0}D}$..

Let us now recall what surface electrons screening does to the point charge
potential in a TI film without gate. We saw in Section II that TI surface
electrons screening length is given by $\lambda=\sqrt{r_{0}/q_{s}}$. Now for a
TI film with the gate we have both gate and surface electron screening working
together. Comparing the expressions of $\Lambda$ and $\lambda$ we see, as one
could expect, that the distance $D(2\kappa/\kappa_{1})\sim D$, which is
essentially the distance to the gate, should play the role of $q_{s}^{-1}$
666The factor $(2\kappa/\kappa_{1})$ is of order unity if
$\kappa_{1}\simeq\kappa_{2}$ are not quite different.. This means that if
$D\gtrsim q_{s}^{-1}$ the gate plays only a perturbative role, while for in
the case $D\lesssim q_{s}^{-1}$ the distance $D$ should replace $q_{s}^{-1}$
in the final result for $\Gamma$. Replacing $q_{s}^{-1}$ by
$D(2\kappa/\kappa_{1})$ in the case of $\lambda\ll r_{0}$ in Eq. (12) yields

$\Gamma=2\quantity(\frac{\pi e^{4}ND}{\kappa_{f}\kappa_{1}})^{1/2}.$ (24)

This result is valid when it is smaller than Eq. (15), i.e. at $D\lesssim
D_{c}=N^{-1/3}\alpha^{-4/3}(\kappa_{1}/\kappa_{f})$.

In most experiments, $D>D_{c}$, so the screening by gate is negligible
compared to surface electrons screening. For example, in Ref. Chong  _et al._
(2020), the gate separation is $D\simeq 20$ nm, while $D_{c}\sim 5$ nm
assuming $\kappa_{f}/\kappa_{1}\sim 10$ and $N\sim 10^{19}$ cm-3.

##  V conductivity

In this section, we calculate the conductivity of the surface for the scenario
(a) in section II assuming that $D>D_{c}$. In the linear screening region
$\mu^{2}\gg e^{2}\phi^{2}$, where the electron density is weakly perturbed by
impurities, using Boltzmann kinetic equation for Dirac electrons, one has the
expression of the conductivity for a single surface Culcer and Winkler (2008),

$\sigma=\frac{e^{2}}{h}\frac{\mu\tau}{4\hbar}.$ (25)

Here $\tau$ is the transport relaxation time whose inverse is given by

$\frac{1}{\tau}=\frac{\alpha\kappa_{f}Ndk_{F}}{\pi\hbar
e^{2}}\int_{0}^{\pi}d\theta v^{2}(q)(1-\cos\theta)\frac{1}{2}(1+\cos\theta),$
(26)

where $v(q)$ is given by Eq. (7) with $q=2k_{F}\sin\theta/2$ and
$q_{s}=k_{F}\alpha\kappa_{f}/\kappa$. The factor $(1+\cos\theta)/2$ in Eq.
(26) arises when the backscattering is suppressed as a consequence of the spin
texture at the Dirac point, as in Weyl semimetals Burkov  _et al._ (2011).
Changing the integral variable from $\theta$ to $q$, Eq. (26) can be rewritten
as

$\frac{1}{\tau}=\frac{4\pi e^{2}\kappa_{f}\alpha Nd}{\kappa^{2}\hbar
k_{F}}\int_{0}^{2k_{F}}\frac{dq}{2k_{F}}\frac{q^{2}\sqrt{1-(q/2k_{F})^{2}}}{\quantity[q(1+qr_{0})+q_{s}]^{2}}$
(27)

Using $x=q/2k_{F}$, the integral in Eq. (27) can be expressed in a
dimensionless form

$I=\int_{0}^{1}dx\frac{x^{2}\sqrt{1-x^{2}}}{[x(1+2k_{F}r_{0}x)+(\alpha
r_{0}/d)]^{2}}.$ (28)

In the scenario (a) we are considering $d>d_{3}=\alpha^{2/3}N^{-1/3}$ and
$d<d_{c}=\alpha^{-4/3}N^{-1/3}$. First inequality means that $k_{F}d>\alpha$,
because $k_{F}>\Gamma/\hbar v_{F}\simeq(\alpha N)^{1/3}$. Therefore we are
interested in Eq. (28) in the limit $k_{F}d\gg\alpha$. In this case
$k_{F}r_{0}\gg\alpha\kappa_{f}/\kappa>1$, so the integral in Eq. (28) is
approximated by

$I\approx\int_{0}^{1}dx\frac{x^{2}\sqrt{1-x^{2}}}{[2k_{F}r_{0}x^{2}+(\alpha
r_{0}/d)]^{2}}$ (29)

The integral kernel peaks at
$x\simeq(\alpha/2k_{F}d)^{1/2}\simeq(2k_{F}\lambda)^{-1}$, which corresponds
to a momentum transfer $q_{\max}\simeq\lambda^{-1}\ll k_{F}$. [Here we used
Eq. (17) for $\lambda$]. The peak value is
$(4q_{s}r_{0})^{-1}=(2k_{F}d\alpha\kappa_{f}^{2}/\kappa^{2})^{-1}$. The width
of the peak is $\Delta x\sim(k_{F}\lambda)^{-1}$. As a result, the integral in
limit $k_{F}d\gg\alpha$ is given by

$I\approx\frac{\pi}{2\sqrt{2}}\frac{\kappa^{2}/\kappa_{f}^{2}}{(k_{F}d)^{3/2}\alpha^{1/2}}.$
(30)

and Eq. (27) is

$\frac{1}{\tau}\approx\sqrt{2}\pi^{2}\frac{\alpha^{1/2}e^{2}N}{\kappa_{f}\hbar
k_{F}^{5/2}d^{1/2}}.$ (31)

Substituting Eq. (31) into Eq. (25) with $k_{F}=\sqrt{4\pi n}$, we have the
conductivity

$\sigma\approx\frac{e^{2}}{h}\frac{2}{\pi^{1/4}}\frac{n^{7/4}d^{1/2}}{\alpha^{3/2}N}.$
(32)

where $d_{3}\lesssim d\lesssim d_{c}$. At $d\sim d_{c}\sim n^{-1/2}/\alpha$ or
$n\sim(\alpha d)^{-2}$, our conductivity Eq. (32) becomes of the order of

$\sigma\sim\frac{e^{2}}{h}\frac{n^{3/2}}{N\alpha^{2}}$ (33)

and with logarithmic accuracy crosses over to the bulk one Skinner and
Shklovskii (2013). At the charge neutrality point $n=n_{p}\sim(\alpha
N)^{2/3}$, we get the minimum conductivity

$\sigma_{\min}\sim(e^{2}/h)(Nd^{3}/\alpha^{2})^{1/6}$ (34)

which is larger than $e^{2}/h$ in the range of its validity $d\gtrsim
d_{3}=\alpha^{2/3}N^{-1/3}$. At $d\sim d_{c}=\alpha^{-4/3}N^{-1/3}$ our
$\sigma_{\min}\sim e^{2}/h\alpha$ and with logarithmic accuracy crosses over
to the bulk one Skinner and Shklovskii (2013).

It is remarkable that due to Rytova-Chaplik-Entin-Kedysh modification of the
Coulomb potential of charged impurity at large range of electron
concentrations $(\alpha N)^{2/3}\lesssim n\lesssim(\alpha d)^{-2}$, not only
$\Gamma$, but also the conductivity Eq. (32), is determined by the long range
potential with $q\sim\lambda^{-1}\ll k_{F}$. Only at large $n$ and $d$ the
conductivity Eq. (33) is determined the large momentum $q\sim k_{F}$
scattering on standard Coulomb potentials of impurities located at distances
smaller than $k_{F}^{-1}$ from the TI film surface Skinner and Shklovskii
(2013).

The condition of validity of the above conductivity theory is that the local
kinetic energy is larger than $\hbar/\tau$. In the worst case when $\mu=0$ and
the local kinetic energy is of the order of $\Gamma(d)$, using Eqs. (15) and
(31), we get $\Gamma\tau/\hbar\sim\alpha^{-1/3}(Nd^{3})^{1/6}>1$, because
$d>\alpha^{2/3}N^{-1/3}$. This justifies our conductivity results.

##  VI Conductivity and hybridization gap

So far we have ignored the the effect of hybridization gap on the conductivity
near neutrality point $\mu=0$. In the absence of disorder potential and the
Fermi level located inside the hybridization gap TI film becomes an insulator.
There is a big interest to realize such an insulator in BSTS system Chong  _et
al._ (2020); Nandi  _et al._ (2018), which is expected to show the quantum
spin Hall effect and a corresponding four probe resistance $h/2e^{2}$ König
_et al._ (2008). Below, we suggest a modification of the theory Nandi  _et
al._ (2018) of disorder effects on conductivity of the very thin TI film near
neutrality point $\mu=0$, using our theory of the random potential developed
in Section II.

We consider the case $\Gamma(d)>\Delta$ so that
$\Gamma_{\Delta}\simeq\Gamma(d)$. Then the potential of charged impurities
$\phi(\mathbf{r})$ bends both bands on each surface up and down with
characteristic scale $a$, creating at the Fermi level large electron and hole
puddles with diameter $\sim a(\Gamma/\Delta)^{4/3}$ Polyakov and Shklovskii
(1995). These puddles are separated by thin insulating stripes of the width
$x=a\Delta/\Gamma$, which form insulating infinite cluster (see Fig. 4 in Ref.
Nandi  _et al._ (2018)) residing at the potential $\phi(\mathbf{r})$
percolation level $\phi(\mathbf{r})=0$.

At low temperatures, this system can conduct only if electrons can easily
tunnel across these insulating stripes. In a relatively thick TI film where
the hybridization gap $\Delta$ is small enough to allow easy tunneling, the
conductivity of surface states is still metallic. Let us find the upper limit
of $\Delta$ for such metallic films, $\Delta_{c}$. The probability of the
Zener-like tunneling across a thin insulating stripe of width $x$ is Nandi
_et al._ (2018)

$\displaystyle P\propto\exp(-\frac{x\Delta}{\hbar
v_{F}})=\exp(-\frac{a\Delta^{2}}{\Gamma\hbar v_{F}}).$ (35)

For the case $\kappa_{f}/\kappa>\alpha^{-1}\gg 1$ studied in Section II using
$a=\lambda$, substituting Eqs. (15) and (17) into Eq. (35), we find that the
critical value of the hybridization gap at which $P$ loses its exponentially
small factor is

$\Delta_{c}(d)=\frac{e^{2}}{\kappa_{f}d}\frac{(Nd^{3})^{1/4}}{\alpha^{1/2}}.$
(36)

Substituting Eq. (20) into Eq. (36) we solved it for the largest thickness
$d_{\max}$ at which TI films can be considered insulating. We got $d_{\max}=5$
nm for $d_{0}=1.2$ nm and $d_{\max}=8$ nm for $d_{0}=1.8$ nm. In both cases
$\Delta_{c}(d_{\max})<\Gamma(d_{\max})$, so that our theory is applicable.

For the STO case studied in Section III,
$\Gamma=\Gamma_{1}(d)\sim(e^{2}/\kappa_{f}d)\sqrt{Nd^{3}}$ and $a=d$.
Substituting these values into Eq. (35) we arrive to the same $\Delta_{c}(d)$
Eq. (36) 777This universal value is larger than the estimate of Ref. Nandi
_et al._ (2018), where the bulk TI surface screening radius Skinner and
Shklovskii (2013) was used for $a$.. Thus, our neutrality point conductivity
is valid if $d>d_{\max}$, while spin Hall effect can be achieved only at
$d<d_{\max}$. This result is in qualitative agreement with results of Ref.
Chong  _et al._ (2020). On the other hand, the fact that in Ref. Nandi  _et
al._ (2018) the TI film thickness $d=4$ nm was not sufficiently small still
remains unexplained.

###### Acknowledgements.

We are grateful to S.K. Chong, V.V. Deshpande, B. Skinner, and D. Weiss and
for useful discussions. Y.H. was partially supported by the William I. Fine
Theoretical Physics Institute.

## References

  * Skinner  _et al._ (2012) B. Skinner, T. Chen, and B. I. Shklovskii, Phys. Rev. Lett. 109, 176801 (2012).
  * Ren  _et al._ (2011) Z. Ren, A. A. Taskin, S. Sasaki, K. Segawa, and Y. Ando, Phys. Rev. B 84, 165311 (2011).
  * Knispel  _et al._ (2017) T. Knispel, W. Jolie, N. Borgwardt, J. Lux, Z. Wang, Y. Ando, A. Rosch, T. Michely, and M. Grüninger, Phys. Rev. B 96, 195135 (2017).
  * Skinner and Shklovskii (2013) B. Skinner and B. I. Shklovskii, Phys. Rev. B 87, 075454 (2013).
  * Skinner  _et al._ (2013) B. Skinner, T. Chen, and B. Shklovskii, Journal of Experimental and Theoretical Physics 117, 579 (2013).
  * Beidenkopf  _et al._ (2011) H. Beidenkopf, P. Roushan, J. Seo, L. Gorman, I. Drozdov, Y. San Hor, R. J. Cava, and A. Yazdani, Nature Physics 7, 939 (2011).
  * Zhang  _et al._ (2010) Y. Zhang, K. He, C.-Z. Chang, C.-L. Song, L.-L. Wang, X. Chen, J.-F. Jia, Z. Fang, X. Dai, W.-Y. Shan, _et al._ , Nature Physics 6, 584 (2010).
  * Kim  _et al._ (2013) D. Kim, P. Syers, N. P. Butch, J. Paglione, and M. S. Fuhrer, Nature communications 4, 2040 (2013).
  * Nandi  _et al._ (2018) D. Nandi, B. Skinner, G. H. Lee, K.-F. Huang, K. Shain, C.-Z. Chang, Y. Ou, S.-P. Lee, J. Ward, J. S. Moodera, P. Kim, B. I. Halperin, and A. Yacoby, Phys. Rev. B 98, 214203 (2018).
  * Chong  _et al._ (2020) S. K. Chong, L. Liu, T. D. Sparks, F. Liu, and V. V. Deshpande, “Topological phase transitions in a hybridized three-dimensional topological insulator,” (2020), arXiv:2004.04870 [cond-mat.mes-hall] .
  * Chaudhuri  _et al._ (2021) D. Chaudhuri, M. Salehi, S. Dasgupta, M. Mondal, J. Moon, D. Jain, S. Oh, and N. P. Armitage, Phys. Rev. B 103, L081110 (2021).
  * Di Bernardo  _et al._ (2021) I. Di Bernardo, J. Hellerstedt, C. Liu, G. Akhgar, W. Wu, S. A. Yang, D. Culcer, S.-K. Mo, S. Adam, M. T. Edmonds, and M. S. Fuhrer, Advanced Materials 33, 2005897 (2021).
  * Wang  _et al._ (2020) J. Wang, C. Gorini, K. Richter, Z. Wang, Y. Ando, and D. Weiss, Nano Letters 20, 8493 (2020).
  * Rytova (1967) N. S. Rytova, Moscow University Physics Bulletin 3, 18 (1967).
  * Chaplik and Entin (1971) A. Chaplik and M. Entin, Journal of Experimental and Theoretical Physics 34, 1335 (1971).
  * Keldysh (1979) L. V. Keldysh, Soviet Journal of Experimental and Theoretical Physics Letters 29, 658 (1979).
  * Laturia  _et al._ (2018) A. Laturia, M. L. Van de Put, and W. G. Vandenberghe, npj 2D Materials and Applications 2, 1 (2018).
  * Note (1) TF approximation is justified as long as $\alpha\ll 1$. See related discussion in Ref. skinner2013a.
  * Note (2) Here we drop the short distance contribution $r<d$ to the potential $\phi(\mathbf{r})$ which is standard Coulomb potential. This does not change the result substantially as long as $d\ll\lambda$.
  * Note (3) The value of $d_{c}$ will be determined after we obtained $\lambda(d)$ self-consistently in Eq. (17).
  * Robertson (2004) J. Robertson, The European physical journal applied physics 28, 265 (2004).
  * Shklovskii (2007) B. I. Shklovskii, Phys. Rev. B 76, 233411 (2007).
  * Note (4) On the other hand, if $N_{e}\lambda/d>N$ the screening length $\lambda$ should be recalculated self-consistently together with $\Gamma$. Then, instead of Eqs. (15) and (17), we arrive at new results $\Gamma\sim(e^{2}N_{e}^{1/3}/\kappa_{f})\alpha^{-6/7}(N_{e}d^{3})^{-1/21}$ and $\lambda\sim d\alpha^{-4/7}(N_{e}d^{3})^{-1/7}$.
  * Kondovych  _et al._ (2017) S. Kondovych, I. Luk’yanchuk, T. I. Baturina, and V. M. Vinokur, Scientific Reports 7, 42770 (2017).
  * Note (5) The length $\Lambda$ can be found also via the following simple variational estimate. Let us assume that electric field lines are confined inside the TI film within radius $\Lambda\ll r_{0}$ from the point charge and exit from the film to the gate in the area $\sim\pi\Lambda^{2}$ . The total electrostatic energy consists of two major contributions, one is the energy of the field inside the film $\sim(e^{2}/\kappa_{f}d)\ln(\Lambda/d)$, and the other one is the field energy in the gate dielectric $\sim e^{2}D/\kappa_{1}\Lambda^{2}$. Minimizing the total energy yields $\Lambda=\sqrt{Dd\kappa_{f}/\kappa_{1}}\sim\sqrt{r_{0}D}$.
  * Note (6) The factor $(2\kappa/\kappa_{1})$ is of order unity if $\kappa_{1}\simeq\kappa_{2}$ are not quite different.
  * Culcer and Winkler (2008) D. Culcer and R. Winkler, Phys. Rev. B 78, 235417 (2008).
  * Burkov  _et al._ (2011) A. A. Burkov, M. D. Hook, and L. Balents, Phys. Rev. B 84, 235126 (2011).
  * König  _et al._ (2008) M. König, H. Buhmann, L. W. Molenkamp, T. Hughes, C.-X. Liu, X.-L. Qi, and S.-C. Zhang, Journal of the Physical Society of Japan 77, 031007 (2008).
  * Polyakov and Shklovskii (1995) D. G. Polyakov and B. I. Shklovskii, Phys. Rev. Lett. 74, 150 (1995).
  * Note (7) This universal value is larger than the estimate of Ref. Nandi  _et al._ (2018), where the bulk TI surface screening radius Skinner and Shklovskii (2013) was used for $a$.