# Human‑in‑the‑loop Machine Translation with Large Language Model

Xinyi Yang11footnotemark: 1<EMAIL_ADDRESS>
Runzhe Zhan<EMAIL_ADDRESS>
Derek F. Wong<EMAIL_ADDRESS>
Junchao Wu<EMAIL_ADDRESS>
Lidia S. Chao<EMAIL_ADDRESS>
NLP2CT Lab, Department of Computer and Information Science, University of
Macau  Equal Contribution. Xinyi Yang contributes to the experiments, data
curation, and analysis. Runzhe Zhan contributes to the methodology, code
skeleton, and paper drafting. Corresponding Author.

###### Abstract

The large language model (LLM) has garnered significant attention due to its
in-context learning mechanisms and emergent capabilities. The research
community has conducted several pilot studies to apply LLMs to machine
translation tasks and evaluate their performance from diverse perspectives.
However, previous research has primarily focused on the LLM itself and has not
explored human intervention in the inference process of LLM. The
characteristics of LLM, such as in-context learning and prompt engineering,
closely mirror human cognitive abilities in language tasks, offering an
intuitive solution for human-in-the-loop generation. In this study, we propose
a human-in-the-loop pipeline that guides LLMs to produce customized outputs
with revision instructions. The pipeline initiates by prompting the LLM to
produce a draft translation, followed by the utilization of automatic
retrieval or human feedback as supervision signals to enhance the LLM’s
translation through in-context learning. The human-machine interactions
generated in this pipeline are also stored in an external database to expand
the in-context retrieval database, enabling us to leverage human supervision
in an offline setting. We evaluate the proposed pipeline using the
GPT-3.5-turbo API on five domain-specific benchmarks for German-English
translation. The results demonstrate the effectiveness of the pipeline in
tailoring in-domain translations and improving translation performance
compared to direct translation instructions. Additionally, we discuss the
experimental results from the following perspectives: 1) the effectiveness of
different in-context retrieval methods; 2) the construction of a retrieval
database under low-resource scenarios; 3) the observed differences across
selected domains; 4) the quantitative analysis of sentence-level and word-
level statistics; and 5) the qualitative analysis of representative
translation cases. The code and data are available at
https://github.com/NLP2CT/HIL-MT/.

Keywords: Machine Translation, Large Language Model, Human-in-the-loop, In-
context Learning, Prompt Engineering, Natural Language Processing

##  1 Introduction

Large language models (LLMs) have exhibited remarkable proficiency in
comprehending natural language prompts (OpenAI,, 2023; Touvron et al.,, 2023),
enabling them to execute various controllable generation tasks based on human
instructions. Furthermore, LLMs can acquire knowledge from limited
demonstrations that are relevant to the input data and generate desired
outputs through analogy. This paradigm, known as in-context learning (ICL)
(Dong et al.,, 2023), represents a significant advancement in prompt
engineering and offers insights into adapting LLMs to downstream tasks without
the need for fine-tuning the models.

Machine translation (MT) serves as a representative sequence-to-sequence task
that also requires tailoring models to produce domain-specific translations.
Traditional approaches to building domain-specific MT models involve fine-
tuning pre-trained models with domain data or utilizing domain adaptation
techniques to transfer in-domain MT models to out-of-domain requirements.
However, these methods are suited for accessible, medium-scale MT models,
which may not be suitable for LLMs. Notably, certain LLMs available through
application programming interfaces (API) lack accessible weight matrices.
Furthermore, optimizing LLM parameters with domain data can be expensive under
resource-limited scenarios. Consequently, current research on LLM-based MT
predominantly focuses on ICL, including in-context selection methods (Agrawal
et al.,, 2022), in-context prompt engineering (Zhang et al.,, 2023), and the
systematic evaluation of LLM-based MT (Hendy et al.,, 2023; Jiao et al.,,
2023). While these lines of research present empirical studies investigating
ICL in the MT task, they still exhibit a dearth of exploration in customizing
LLMs for domain-specific needs. Given the representative capabilities of LLMs,
which rely on generating outputs based on provided instructions, it is
intuitive to leverage LLMs to refine general MT outputs for different domains.

Nevertheless, there remain challenges and issues when utilizing ICL to adapt
LLMs for domain-specific needs. Firstly, ICL demonstrations for MT typically
comprise source input and target reference, lacking domain features. Secondly,
the ICL retrieval database is usually constructed using separate labeled data,
and the retrieved demonstrations fail to capture LLMs’ translation
preferences. Last but not least, the adaptation of black-box LLMs does not
benefit from parameter optimization, thereby limiting adaptation methods to
modifying ICL inputs alone. In response to these challenges, we propose
integrating LLM-specific translation feedback into ICL inputs, enabling the
model to learn from both relevant input-output pairs and domain preferences.

Specifically, the proposed pipeline consists of two essential parts: feedback
collection, and in-context refinement. To collect LLM-specific feedback
associated with domains, we first request the LLM to produce domain-specific
translations and obtain feedback by comparing its translation with a reference
translation. The feedback takes the form of a sequence of revision
instructions, indicating the necessary edits to transform the LLM’s
translation into the reference translation. Ideally, these revision
instructions originate from human feedback sources. However, due to resource
limitations, we simulate this process and generate synthetic human feedback in
this study. Subsequently, these translation texts and feedback are stored
together in the ICL retrieval database. For in-context refinement, when faced
with new in-domain translation requests, the pipeline initially prompts the
LLM to generate a draft translation. It then retrieves similar translation
pairs and their revision histories as in-context demonstrations tied to the
specific domain. Finally, the model refines the draft translation based on the
retrieved domain-specific demonstrations. Any new human-machine interactions
generated within this pipeline are incorporated to expand the in-context
retrieval database. Overall, the primary concept revolves around enabling the
LLM to revise its outputs by learning from relevant domain-specific revision
feedback.

We conduct experiments using the proposed pipeline in the German-English
translation direction across five domains, utilizing the GPT-3.5 Turbo API as
a testbed for the black-box LLM. The results demonstrate that the proposed
pipeline enhances domain translation performance in selected domains, as
indicated by four automated evaluation metrics. To further elucidate the
effectiveness of our approach, we discuss the pipeline and results through an
ablation study, quantitative analysis, and qualitative analysis.

##  2 Methodology

Figure 1: Illustration of the proposed human-in-the-loop translation method in
the context of the large language model.

We leave the discussion of related work in the Appendix A due to page
limitation. The overall pipeline is illustrated in Figure 1. The feedback
retrieved from a data store aims to provide domain-specific revision
demonstrations to LLM. Furthermore, it can also be jointly used with external
human supervision in real-world applications. In addition to the feedback that
exists in the database, any newly produced feedback for current text will be
also recorded into the database to be used as a candidate for ICL retrieval in
the future.

###  2.1 Feedback Collection

To correct LLM’s bias in domain translation through the ICL paradigm, we must
first construct an ICL retrieval database that reflects the gap between LLM’s
translation preferences and domain preferences. To do this, we ask LLM to
translate several domain texts first, and then use automated methods or human
intervention to generate feedback on LLM’s translation. To simulate the
process of human feedback, we use an automated evaluation method based on edit
distance theory to generate feedback for the translations. Computing edit
distance is a dynamic programming problem, where the cost matrix reflects what
editing operations are needed to transform from the machine translation to the
reference translation. Specifically, bottom-up recursion yields the minimum
cost of editing the translation, so we can generate human-like feedback by
back-tracing the optimal alignment of the cost matrix and converting it to
natural language. There are three kinds of editing operations are considered
in our case: deletion, insertion, and substitution. Given an LLM’s translation
$h$ and reference translation $r$, the cost matrix $D(i,j)$ indicates the edit
distance between $h_{<i}$ and $r_{<j}$. Let the cost of deletion, insertion,
and substitution be 1, the cost matrix $D(i,j)$ can be calculated as:

$\displaystyle D(i,0)$ $\displaystyle=i$ (1) $\displaystyle D(0,j)$
$\displaystyle=j$ $\displaystyle D(i,j)$
$\displaystyle={\begin{cases}D(i-1,j-1)&{\text{if}}\;h_{i}=r_{j}\\\
\min{\begin{cases}D(i-1,j)+1\\\ D(i,j-1)+1\\\
D(i-1,j-1)+1\end{cases}}&{\text{if}}\;h_{i}\neq r_{j}\end{cases}}$

We can obtain an optimal path of minimum edit distance by back-tracing the
cost matrix $D$ and then convert this path to the natural language instruction
as shown in Figure 2. Once the feedback of a specific test instance is
produced, we combine it with the source text, LLM’s translation, and reference
translation to form the new ICL demonstration instance. This enables LLM to
learn from its deviations from in-domain reference translations and how to
polish the draft translation.

Figure 2: An example of describing the optimal path of minimum edit distance.

###  2.2 In-context refinement

We construct a database for ICL retrieval using the automatic feedback
generation method mentioned in the previous section. In this section, we will
describe how to retrieve and utilize these ICL demonstrations as well as
feedback records for conducting a two-stage translation.

#### Demonstration Retrieval

We retrieve the relevant ICL demonstrations by evaluating the relevance
between test instances and examples stored in the database. Specifically, we
follow the previous exploratory work on ICL for MT and use two retrieval
metrics: the BM25 score (Robertson et al.,, 1995) and the BM25-rerank score
(Agrawal et al.,, 2022). BM25 is a common retrieval metric that evaluates the
relevance between the query and the documents. Here we consider each source
sentence in the ICL database as a document and calculate the BM25 score.
BM25-Rerank, as a post-screening method for BM25, first selects the $N$
samples with the highest BM25 scores and re-scores these samples based on the
n-gram recall score $R$ to select the top-$K$ samples as ICL demonstrations.
Let the input in source language be $s$, and a demonstration candidate stored
in the database be $c$, then the recall score can be calculated as:

$R=\exp{\frac{1}{n}\sum_{i=1}^{n}\log\frac{\text{Count}(\text{$i$-gram}\in
s\cap c)}{\text{Count}(\text{$i$-gram}\in s)}}$ (2)

#### Two-stage Translation

As mentioned in the previous section, we first ask the LLM to generate a draft
translation, and then polish the draft translation by providing the ICL
demonstrations, resulting in a two-stage translation process. In practice, we
implement the two-stage translation through the multi-turn dialog feature of
the GPT-3.5 API. In addition, we empirically found that the LLM may
incorrectly modify the draft translation after observing the ICL
demonstrations. We attribute this phenomenon to the fact that the scale
limitation of the retrieval database, which may lead to the inclusion of some
irrelevant samples. Therefore, we asked the LLM further compare the polished
translations with the draft translations at the second stage, and finally
select the higher quality one. It is worth noting that the LLM did not observe
the reference translation of the test instance during the whole process but
only compared the translation quality by means of self-reflection.

##  3 Experiments

###  3.1 Data and Evaluation

We verify the effectiveness of the proposed pipeline on a multi-domain German-
English translation benchmark (Aharoni and Goldberg,, 2020). The test data
keeps the same setting as the same benchmark, which involves five domains
including IT, Koran, Law, Medical, and Subtitles. We randomly sample 2,000
samples from the training set of each domain as the source of constructing the
ICL retrieval database. To complete this process, we use GPT-3.5 API to
generate the translation and employ the edit-distance-based method mentioned
in Section 2.1 to produce human-like feedback. We use several automated
metrics to evaluate the translation quality, including BLEU (Papineni et al.,,
2002), TER (Snover et al.,, 2006), BERTScore (Zhang et al.,, 2020), and COMET
(Rei et al.,, 2020). BLEU and TER are the traditional metrics that evaluate
the text overlap whereas the others can evaluate the semantic overlap based on
neural networks. We also found that APIs sometimes produce hallucinations or
refuse to translate some sentences. To make a fair comparison, we manually
check the translation results and remove the invalid results, and only
evaluate the performance of the sentences that are successfully translated by
all the methods.

###  3.2 Settings

The experiments were conducted using GPT-3.5 API. The decoding temperature and
the top_p parameters are set to $1$ by default. When evaluating the relevance
of demonstrations, we first retrieve the top $K$=200 demonstrations with the
highest BM25 scores and then select top-$N$ demonstrations with the highest
$4$-gram re-rank scores as the finalized ICL demonstrations.

###  3.3 Main Results

Table 1 presents the automated evaluation results of the baseline methods. The
experimental findings demonstrate that the proposed approach effectively
enhances the performance of GPT-3.5-Turbo baseline. Importantly, we observe
that the impact of the proposed HIL method varies across different domains, a
topic that will be thoroughly examined in Section 4.2. In addition, we were
limited to varying the number of ICL demonstrations from 1 to 3 due to
constraints on request tokens. Nonetheless, the results strongly indicate that
providing more ICL demonstrations leads to improved performance. Moreover,
when evaluating the performance with neural metrics, the differences in scores
are not substantial compared to the traditional metrics. We postulate that the
process of refining the draft translation may not deviate significantly from
the original semantic content but rather brings it closer to specific
translation preferences in certain domains. These findings will be elucidated
through a detailed case study in the subsequent section.

| IT | Koran
---|---|---
| BLEU | TER | BERT-F | COMET | BLEU | TER | BERT-F | COMET
GPT-3.5-Turbo | 34.4 | 62.3 | 93.2 | 82.5 | 16.2 | 74.1 | 90.6 | 73.4
+1-shot HIL | 29.0 | 77.3 | 92.5 | 81.1 | 15.7 | 80.8 | 90.5 | 72.8
+2-shot HIL | 33.9 | 64.7 | 92.9 | 82.3 | 15.8 | 77.7 | 90.6 | 73.2
+3-shot HIL | 32.6 | 68.8 | 93.0 | 82.2 | 16.5 | 76.0 | 90.7 | 73.6
+Compare HIL | 35.2 | 61.3 | 93.3 | 82.8 | 16.6 | 74.6 | 90.7 | 73.8
| Law | Medical
GPT-3.5-Turbo | 37.6 | 54.7 | 93.7 | 83.8 | 40.0 | 59.4 | 93.9 | 83.4
+1-shot HIL | 36.2 | 59.6 | 93.5 | 83.1 | 36.6 | 67.9 | 93.3 | 82.1
+2-shot HIL | 37.0 | 56.6 | 93.6 | 83.5 | 39.2 | 63.0 | 93.6 | 83.0
+3-shot HIL | 36.7 | 56.5 | 93.6 | 83.7 | 38.4 | 63.0 | 93.7 | 83.2
+Compare HIL | 37.7 | 54.5 | 93.8 | 83.9 | 40.9 | 60.1 | 93.9 | 83.6
| Subtitles |  |  |  |
GPT-3.5-Turbo | 27.9 | 64.8 | 93.0 | 80.0 |  |  |  |
+1-shot HIL | 26.3 | 69.4 | 92.5 | 78.8 |  |  |  |
+2-shot HIL | 26.4 | 67.1 | 92.6 | 79.4 |  |  |  |
+3-shot HIL | 27.4 | 64.6 | 93.0 | 79.8 |  |  |  |
+Compare HIL | 28.0 | 64.1 | 93.1 | 80.1 |  |  |  |

Table 1: Automated evaluation results of different translation strategies.
“$K$-shot HIL” indicates the proposed HIL method with $K$ demonstrations used.
“Compare HIL” indicates using the comparison strategy to finalize the two-
stage translation.

##  4 Analysis

Method | BM25 | BM25 Re-Rank |
---|---|---|---
BLEU | TER | BERT-F | COMET | BLEU | TER | BERT-F | COMET |
IT | 34.9 | 62.3 | 93.1 | 82.5 | 35.2 | 61.3 | 93.3 | 82.8 |
Koran | 16.2 | 74.3 | 90.7 | 73.7 | 16.6 | 74.6 | 90.7 | 73.8 |
Law | 38.0 | 54.2 | 93.8 | 84.2 | 37.7 | 54.5 | 93.8 | 83.9 |
Medical | 40.6 | 58.8 | 94.0 | 83.8 | 40.9 | 60.1 | 94.0 | 83.6 |
Subtitles | 28.2 | 64.5 | 93.0 | 80.2 | 28.0 | 64.1 | 93.1 | 80.1 |

Table 2: Automated evaluation results for 3-shot HIL with different ICL
retrieval strategies.

###  4.1 Effects of ICL Retrieval Methods

To explore the potential impact of different demonstration retrieval methods
on our proposed HIL translation workflow, we conducted experiments using two
strategies: BM25 and BM25 Re-rank, in a 3-shot translation scenario. The
comparative results are summarized in Table 2. Based on the automated metrics,
both BM25 and BM25 Re-Rank methods exhibited strengths and weaknesses in
various domains. Specifically, BM25 Re-Rank slightly outperformed BM25 in
terms of the BLEU metric. The advantage of BM25 Re-Rank was particularly
evident in the IT domain, as it achieved higher scores than BM25 across all
metrics. However, this conclusion was reversed in the Law domain. The
advantage of the BM25 Re-Rank strategy lies in its ability to filter out
repetitive context with identical queries. Consequently, this approach selects
more relevant examples related to the IT domain, leading to an improvement in
the quality of the translation output within this domain. The BM25 method for
demonstration retrieval focuses on document frequency and keyword matching,
making it more effective in ensuring proper usage of legal terminology. This
observation underscores the necessity of adopting different strategies for
demonstration retrieval across different domains to ensure the selection of
contextually relevant and domain-specific examples for the target sentences.

###  4.2 Domain Differences

In general, the HIL approach exhibits superior performance compared to the
GPT-3.5 API baseline across all five domains, with particularly notable
advantages in the IT and Medical domains. However, the differences in
performance are relatively smaller in Law and Subtitles domains. These
variations can be attributed to the distinct sentence styles and structures
prevalent in each domain. Upon analyzing the translation results, it becomes
evident that HIL excels in IT and Medical domains by effectively aligning
terminology with the reference translations. For example, consider the phrase
“Returns a character string” in the IT domain, HIL correctly recognizes the
need to use the third-person singular form of the word “returns” and avoids
translating “character string” simply as “string”. While these words may not
be technical terms, their specific usage preferences in the IT domain are
crucial, and such nuances cannot be captured by the GPT-3.5 API baseline.
Conversely, in domains such as Law and Subtitles, where HIL’s performance is
comparatively lower, the sentences tend to adhere to specific legal clauses or
follow a more colloquial and concise style. As GPT-3.5 is a multi-domain
language model, it already possesses substantial knowledge related to these
domains, leading to satisfactory draft translations in the initial output,
thereby reducing the necessity for extensive corrections through
demonstrations. Furthermore, it is worth noting that the quality of the data
within the demonstration pool may not be very high, as they were randomly
sampled from the training set. This behavior could also have a negative impact
on the final results.

###  4.3 Quantitative Analysis

We conducted quantitative analysis on the translation results with the help of
compare-mt111https://github.com/neulab/compare-mt (Neubig et al.,, 2019)
toolkit.

#### Part of Speech (POS)

We applied Stanford’s POS tagging toolkit (Toutanova et al.,, 2003) to label
the target-side text and subsequently examined the translation outcomes of
both the baselines and HIL method across different POS categories. The results
are presented in Figure 3. Overall, HIL outperforms the baselines in all noun
categories, including singular or mass nouns (NN), plural nouns (NNS), and
singular proper nouns (NNP). Additionally, among the three verb types, HIL
exhibited superior performance to the baseline in base form verbs (VB) and
third person singular present verbs (VBZ), with the most noticeable advantage
observed in base form verbs (VB). Furthermore, the advantage of HIL is
particularly evident in POS categories where the baseline exhibits lower
translation accuracy. These findings underscore the effectiveness of the HIL
approach in handling various parts of speech and its ability to deliver
improved translation outputs, especially in challenging linguistic contexts.

CCDTINJJNNNNPNNSPRPRBTOVBVBPVBZother$0.4$$0.5$$0.6$$0.7$POS
TaggingAccuracyDraftHIL Figure 3: The translation comparison in terms of
different POS tags.

#### Sentence Length

Figure 4 presents the translation performance of both the baseline and HIL
models for sentences of varying lengths. HIL exhibits a clear advantage over
the baseline for sentences that are shorter than 10 words as well as those
longer than 60 words. Unfortunately, HIL performs less effectively for medium-
length sentences compared to the baseline’s initial draft. This observation
indicates that there might be challenges specific to this sentence length
range that warrant further investigation and potential refinement of the HIL
approach.
In conclusion, our analysis of translation performance across different POS
categories and sentence lengths reveals that HIL exhibits exceptional
proficiency in translating both nouns and verbs, particularly excelling in
extreme-short and extreme-long sentences.

IT Domain
---
Source |  Das Handbuch zu & ksnapshot;
Reference |  The & ksnapshot; Handbook
Demonstrations |  1\. <input>Das Handbuch zu & kontact; <hypothesis>The manual for & kontact; <reference>& kontact; Handbook <revision>“the” should be deleted. “manual” should be deleted. “for” should be deleted.”, “handbook” should be inserted after “kontact;”
|  2\. <input>Das Handbuch zu & kanagram; <hypothesis>The manual for &
kanagram; <reference>& kanagram; Handbook <revision>“the” should be deleted.
“manual” should be deleted. “for” should be deleted. “handbook” should be
inserted after “kanagramt;”
|  3\. …
Draft |  The manual for & ksnapshot;
ICL |  The manual for &ksnapshot;
HIL |  The & ksnapshot; Handbook
Table 3: An example result of three different translation strategies. “Draft”
represents the preliminary translation results obtained at the initial turn in
our HIL pipeline. “ICL” presents the translation results achieved using
ordinary ICL demonstrations without revision feedback.
$<$10[10,20)[20,30)[30,40)[40,50)[50,60)$\geq$60$20$$22$$24$$26$$28$$30$Sentence
LengthBLEU ScoreDraftHIL Figure 4: The translation comparison across various
sentence lengths.

###  4.4 Case Study

Below, we present an illustrative example from the IT domain to showcase the
advantages of HIL in terms of terminology translation. Additionally, we
compare the proposed HIL method with the ordinary ICL method without revision
feedback. The results are shown in Table 3. In the given example, while the
German word “Handbuch” was translated as “manual” in both the draft and ICL
translations, the reference and retrieved demonstration suggest that
“Handbook” is a more accurate translation. Regarding the terminology usage of
the IT domain, “Handbook” precisely describes a professional document type,
whereas “manual” might be more generic and provide less specific information.
Among the three translation methods, the HIL translation successfully captures
this crucial information based on the provided demonstrations. On the other
hand, HIL also learns from revisions in the demonstration and opts to
translate the original sentence as “The &ksnapshot; Handbook”, aligning more
accurately with the word order of the original text. This demonstrates how HIL
can effectively incorporate valuable revision feedback to produce more
contextually appropriate and accurate translations.

##  5 Conclusions and Future Work

In this paper, we present an empirical study focused on enhancing the
translation capabilities of the LLM by integrating concrete feedback within
the translation process. Our objective is to establish a human-in-the-loop
machine translation pipeline, where human feedback plays a pivotal role. To
simulate this concept, we utilize an automated feedback method, leveraging the
GPT-3.5 API as our testbed, which yields effective results. In the future, our
plan is to collect human feedback to create a novel dataset and conduct
experiments on the proposed pipeline using this dataset. This approach will
enable us to further validate and implement our human-in-the-loop machine
translation system in the context of LLM, enhancing its practical
applicability and performance.

## Acknowledgement

This work was supported in part by the Science and Technology Development
Fund, Macau SAR (Grant Nos. FDCT/0070/2022/AMJ, FDCT/060/2022/AFJ), the Multi-
year Research Grant from the University of Macau (Grant No.
MYRG2020-00054-FST), and the Research Program of Guangdong Province (Grant No.
2220004002576). This work was performed in part at SICC which is supported by
SKL-IOTSC, and HPCC supported by ICTO of the University of Macau.

## References

  * Agrawal et al., (2022) Agrawal, S., Zhou, C., Lewis, M., Zettlemoyer, L., and Ghazvininejad, M. (2022).  In-context examples selection for machine translation.  ArXiv preprint, abs/2212.02437.
  * Aharoni and Goldberg, (2020) Aharoni, R. and Goldberg, Y. (2020).  Unsupervised domain clusters in pretrained language models.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7747–7763. Association for Computational Linguistics.
  * Dong et al., (2023) Dong, Q., Li, L., Dai, D., Zheng, C., Wu, Z., Chang, B., Sun, X., Xu, J., and Sui, Z. (2023).  A survey for in-context learning.  ArXiv preprint, abs/2301.00234.
  * Hendy et al., (2023) Hendy, A., Abdelrehim, M. G., Sharaf, A., Raunak, V., Gabr, M., Matsushita, H., Kim, Y. J., Afify, M., and Awadalla, H. H. (2023).  How good are gpt models at machine translation? a comprehensive evaluation.  ArXiv preprint, abs/2302.09210.
  * Jiao et al., (2023) Jiao, W., Wang, W., tse Huang, J., Wang, X., and Tu, Z. (2023).  Is chatgpt a good translator? yes with gpt-4 as the engine.  volume abs/2301.08745.
  * Neubig et al., (2019) Neubig, G., Dou, Z.-Y., Hu, J., Michel, P., Pruthi, D., and Wang, X. (2019).  compare-mt: A tool for holistic comparison of language generation systems.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 35–41. Association for Computational Linguistics.
  * OpenAI, (2023) OpenAI (2023).  Gpt-4 technical report.  ArXiv preprint, abs/2303.08774.
  * Papineni et al., (2002) Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002).  Bleu: a method for automatic evaluation of machine translation.  In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318. Association for Computational Linguistics.
  * Rei et al., (2020) Rei, R., Stewart, C., Farinha, A. C., and Lavie, A. (2020).  COMET: A neural framework for MT evaluation.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702. Association for Computational Linguistics.
  * Robertson et al., (1995) Robertson, S. E., Walker, S., Jones, S., Hancock-Beaulieu, M. M., Gatford, M., et al. (1995).  Okapi at trec-3.  Nist Special Publication Sp, 109:109.
  * Snover et al., (2006) Snover, M., Dorr, B., Schwartz, R., Micciulla, L., and Makhoul, J. (2006).  A study of translation edit rate with targeted human annotation.  In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers, pages 223–231. Association for Machine Translation in the Americas.
  * Toutanova et al., (2003) Toutanova, K., Klein, D., Manning, C. D., and Singer, Y. (2003).  Feature-rich part-of-speech tagging with a cyclic dependency network.  In Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, pages 252–259.
  * Touvron et al., (2023) Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. (2023).  Llama: Open and efficient foundation language models.  ArXiv preprint, abs/2302.13971.
  * Wang et al., (2022) Wang, D., Wei, H., Zhang, Z., Huang, S., Xie, J., and Chen, J. (2022).  Non-parametric online learning from human feedback for neural machine translation.  In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22 - March 1, 2022, pages 11431–11439. AAAI Press.
  * Wu et al., (2022) Wu, X., Xiao, L., Sun, Y., Zhang, J., Ma, T., and He, L. (2022).  A survey of human-in-the-loop for machine learning.  Future Generation Computer Systems, 135:364–381.
  * Zhang et al., (2023) Zhang, B., Haddow, B., and Birch, A. (2023).  Prompting large language model for machine translation: A case study.  ArXiv preprint, abs/2301.07069.
  * Zhang et al., (2020) Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., and Artzi, Y. (2020).  Bertscore: Evaluating text generation with BERT.  In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.

##  Appendix A Appendix: Related Work

###  A.1 ICL-based MT

Agrawal et al., (2022) proposed an ICL example selection method for machine
translation, aiming to explore the impact of ICL examples on translation
output quality. To address the issues in existing BM25 retrieval methods, the
proposed approach re-ranks the top 100 candidate sentences selected by the
BM25 score and introduces a re-ranking score, thereby maximizing the coverage
of input words. Zhang et al., (2023) aims to explore different prompts in the
context of LLM-based machine translation. In comparison to previous related
research, the main innovation of this study lies in exploring how to design
prompts for LLMs to enhance their translation capabilities from three
different perspectives. Specifically, the research investigates different
prompting strategies, the utilization of unlabeled data, and the flexibility
of transfer learning. These research achievements demonstrate the significant
value of ICL in improving the quality of machine translation systems.

###  A.2 Human-in-the-Loop MT

The concept of Human-in-the-Loop (HIL) (Wu et al.,, 2022) aims to leverage
user feedback for optimizing the model. Building on this idea, Wang et al.,
(2022) proposes a novel non-parametric online learning method called kNN-over-
kNN (KoK) that does not alter the model structure. KoK is a plug-and-play non-
parametric approach that learns online based on human feedback, reducing the
number of user interactions and improving machine translation model
performance. The online learning process of KoK involves three steps:
decoding, correcting, and adapting. In the decoding phase, the MT system
translates the source sentence, and the output is obtained by weighting the
KoK method and kNN-MT. In the correcting phase, users provide corrections of
the machine-translated text, resulting in the post-edited translation.
Finally, in the adapting phase, post-edited and source sentences are jointly
used to expand the data repositories of token-kNN and policy-kNN, thereby
optimizing the model. Through these three steps, the KoK framework can
promptly influence the kNN translation model’s decision-making. It is worth
noting that, as of now, the HIL method has not been applied to LLM
translation. Our paper focuses on exploring the potential integration of HIL
into LLM translation to further enhance its performance and capabilities.


# Energy resonance transfer between quantum defects in metal halide
perovskites

Yu Cui  Xiao-Yi Liu  Shi-Yuan Ji  Yong Sun  Jia-Pei Deng  Xu-Fei Ma  Zhi-Qing
Li  Zi-Wu Wang  Tianjin Key Laboratory of Low Dimensional Materials Physics
and Preparing Technology, Department of Physics, School of Science, Tianjin
University, Tianjin 300354, China<EMAIL_ADDRESS>
###### Abstract

Quantum defects have shown to play an essential role for the non-radiative
recombination in metal halide perovskites (MHPs). Nonetheless, the processes
of charge transfer-assisted by defects are still ambiguous. Herein, we
theoretically study the non-radiative multiphonon processes among different
types of quantum defects in MHPs using Markvart’s model for the induced
mechanisms of electron-electron and electron-phonon interactions,
respectively. We find that charge carrier can transfer between the neighboring
levels of the same type shallow defects by multiphonon processes, but it will
be distinctly suppressed with the increasing of the defect depth. For the non-
radiation multiphonon transitions between donor- and acceptor-like defects,
the processes are very fast and independence of the defect depth, which
provide a possible explanation for the blinking phenomena of photoluminescence
spectra in recent experiment. We also discuss the temperature dependence of
these multiphonon processes and find that their variational trends depend on
the comparison of Huang-Rhys factor with the emitted phonon number. These
theoretical results fill some gaps of defect-assisted non-radiative processes
in the perovskites materials.

##  I Introduction

Metal halide perovskites (MHPs) and their low-dimensional structures are
triggering widespread attention owing to a range of fascinating properties,
such as high quantum yield, remarkable photovoltaic conversion efficiency,
high optical absorption coefficient, and ease of fabricationwx1 ; wx2 ; wx3 ;
wx4 ; wx5 . These properties make MHPs candidates for the next generation of
optoelectronic and photovoltaic devices. However, the inevitable non-radiative
recombination processes, dominating the charge carriers loss, play the crucial
roles to determine their basic properties and device performanceswx6 ; wx7 ;
wx8 ; wx9 ; wx10 . There, for example, are trap states which strongly interact
with the photogenerated charge carriers, making the conversion efficiencies of
perovskite solar cells have not yet reached the thermodynamic limitswx11 ;
wx12 .

Figure 1:  The schematic diagrams of the non-radiative multiphonon processes
of energy resonance transfer between quantum defects in CH3NH3PbI3 MHPs. (a)
represents a quantum transition induced by electron-electron interaction from
the initial state to the final state with its transition energy to be
compensated by multiphonon emission $p\hbar{\omega_{\nu}}$, where the initial
(final) state consists of two electrons located in the defect level $E_{1}$
($E_{2}$) and in the valence band edge $E_{V}$, respectively. $\Delta
E_{1}=|E_{C}-E_{1}|$ is the initial defect depth, $\Delta E_{2}=|E_{C}-E_{2}|$
is the final defect depth, and $E_{C}$ denotes the conduct band edge. (b)
represents the electron transfer induced by electron-phonon interaction, where
the electron relaxes from the initial defect depth $\Delta E_{1}$ to deeper
defect depth $\Delta E_{2}$ directly by the successive emission of phonons
$p\hbar{\omega_{\nu}}$ and $\hbar{\omega_{\nu}}$ denotes the phonon energy.

Without a doubt, an essential condition for perovskites-based devices to reach
higher performances is the elimination of all loss channels, so the studies of
non-radiative loss channels in MHPs are the key topics.

A most concerned non-radiative loss channel is defect-assisted recombination.
In the past years, there have been several correlative proofs about defect-
assisted processes. de Quilettes $el$ $al.$ have observed the defect states
quenched the crystalline fluorescence of discontinuous perovskite films by
measuring the microscopic photoluminescenceb1 . From time-resolved
photoluminescence spectroscopy, Yamada $et$ $al.$ concluded that the defect-
assisted recombination includes non-radiative processes in MHPswx defect6 .
Bolink $et$ $al.$ also found that the actual devices are hampered by the
defect-assisted non-radiative processes according to the voltage dependence of
the electroluminescence efficiency and the ideality factor measurementsb2 . In
fact, defects (or impurities), especially for the deeper defect levels serving
as the trapping centers, play the key role in determining the non-radiative
recombination, because of the requirement for higher thermal energy to leave
from the trapsb5 ; b6 ; b7 , in a way that is ultimately detrimental to the
charge transport and opto-electronic propertiesb8 ; b9 ; b10 ; b11 .
Therefore, the comprehensive understanding of the quantum processes assisted
by defects are of crucial for the fundamental properties of MHPs and their
potential applications. However, the detail studies on the charge carriers of
non-radiative transitions among defect states in these perovskite materials
are very few until now.

In the present paper, we study the non-radiative multiphonon processes of
energy resonance transfer (ERT) between quantum defects based on Markvart’s
modelwx moxing1 ; wx moxing2 ; wx moxing3 ; wx moxing4 . The processes between
the same types of defects, including donor-donor, acceptor-acceptor and
neutral-neutral, as well as between different types of defects, i.e., donor-
acceptor pairs, are taken into account. We present the influence of different
defect depths on the lifetime of these energy transfer processes induced by
the mechanisms of the electron-electron and electron-phonon interactions,
respectively, and give the comparisons between them. We also discuss the
temperature dependences of these multiphonon transitions among different
defects.

##  II theoretical model

The processes of the non-radiative multiphonon ERT among quantum defects for
the induced mechanisms of electron-electron and electron-phonon interactions
are schemed in Fig. 1(a) and (b), respectively. According to the approximation
derived by Markvartwx moxing1 ; wx moxing2 ; wx moxing3 ; wx moxing4 , the
transition rate of multiphonon ERT processes $\tau^{-1}$ can be expressed as

$\displaystyle{\tau_{e-e}^{{\rm{-1}}}}$ $\displaystyle=$
$\displaystyle\frac{{{{\left|{{\mathcal{H}_{ij}^{e-e}}}\right|}^{2}}\sqrt{2\pi}}}{{\hbar\left({p\hbar{\omega_{LO}}}\right)\sqrt{p\sqrt{1+{\chi^{2}_{\lambda}}}}}}\exp[p(\frac{{\hbar{\omega_{LO}}}}{{2k_{B}T}}+\sqrt{1+{\chi^{2}_{\lambda}}}$
(1)
$\displaystyle-\chi_{\lambda}\cosh(\frac{{\hbar{\omega_{LO}}}}{{2k_{B}T}})-\ln(\frac{{1+\sqrt{1+{\chi^{2}_{\lambda}}}}}{\chi_{\lambda}}))],$

with

$\displaystyle{\left|{{\cal
H}_{ij}^{e-e}}\right|^{2}}={\left|{\left\langle{{\phi_{0}}|\langle{\varphi_{i}}\left|{{H_{e-e}}}\right|{\varphi_{j}}\rangle|{\phi_{0}}}\right\rangle}\right|^{2}},$
(2)
$\displaystyle{H_{e-e}}=\frac{{{e^{2}}}}{{4\pi{\varepsilon_{0}}{\varepsilon_{r}}\left|{{{\bf{r}}_{\bf{1}}}{\bf{-}}{{\bf{r}}_{\bf{2}}}}\right|}},$
(3)

for the electron-electron interaction and

$\displaystyle{\tau_{e-p}^{{\rm{-1}}}}$ $\displaystyle=$
$\displaystyle\frac{{{{\left|{{\mathcal{H}_{ij}^{e-p}}}\right|}^{2}}\sqrt{2\pi}}}{{\hbar\left({p\hbar{\omega_{LO}}}\right)\sqrt{p\sqrt{1+{\chi^{2}_{\lambda}}}}}}\exp[p(\frac{{\hbar{\omega_{LO}}}}{{2k_{B}T}}+\sqrt{1+{\chi^{2}_{\lambda}}}$
(4)
$\displaystyle-\chi_{\lambda}\cosh(\frac{{\hbar{\omega_{LO}}}}{{2k_{B}T}})-\ln(\frac{{1+\sqrt{1+{\chi^{2}_{\lambda}}}}}{\chi_{\lambda}}))],$

with

$\displaystyle{{\left|{{\mathcal{H}_{ij}^{e-p}}}\right|}^{2}}={\left|{\left\langle{\varphi_{i}\left|{{H_{e-p}}}\right|{\varphi_{j}}}\right\rangle}\right|^{2}},$
(5) $\displaystyle{H_{e-p}}=\sum\limits_{\bf{q}}{{\cal
M}(q)(a_{-{\bf{q}}}^{\dagger}+{a_{\bf{q}}}){e^{i{\bf{q}}\cdot{\bf{r}}}}},$ (6)
$\displaystyle{\cal
M}(q)=-i\frac{{\hbar{\omega_{LO}}}}{q}{(\frac{{4\pi\alpha}}{V})^{\frac{1}{2}}}{(\frac{\hbar}{{2{m^{*}}{\omega_{LO}}}})^{\frac{1}{4}}},$
(7)

for the electron-phonon interaction.

Figure 2:  Huang-Rhys factors as functions of the initial defect depths ($\Delta E_{1}$) and the final defect depths ($\Delta E_{2}$). (a), (b), and (c) present the donor-donor, acceptor-acceptor, and donor-acceptor processes, respectively. Table 1:  The adopted parameters for the theoretical calculation in MAPbI3.  Parameter | Symbol | Value
---|---|---
Effective Bohr radius | $a^{*}$ | 8.9 nmwx canshu1 ; wx moxing1
Effective mass | $m^{*}$ | 0.2wx canshu1
Phonon energy | $\hbar{\omega_{LO}}$ | 16.5 meVwx canshu1 ; wx moxing1
Fröhlich coupling constant | $\alpha$ | 1.73wx canshu1 ; wx moxing1
| Radius of sphere with Brillouin
---
zone volume
$q_{c}$ | 6.2 /nmwx canshu1
Lattice constant | $a_{0}$ | 0.63 nmwx canshu2
Vacuum permittivity | ${\varepsilon_{0}}$ | 8.85 $\times{10^{-12}}$ F/mwx canshu1
Free electron (hole) mass | ${m_{0}}$ |  $9.1\times{10^{-31}}$ kgwx canshu1
Band gap | $\left|{{E_{C}}-{E_{V}}}\right|$ | 1.6 eVwx moxing1
Quantum defect parameter | $\vartheta$ | Variable
Depth of defect | $\Delta E$ | Variable
Relative permittivity | ${\varepsilon_{r}}$ | 20${\varepsilon_{0}}$

Longitudinal optical (LO) phonon is mainly taken into account because of the
strong charge carrier-LO phonon coupling in MHPswx ep1 ; wx ep2 . $p$ is the
emitted phonon number, $\hbar{\omega_{LO}}$ is the LO phonon energy, $T$ is
the temperature and $k_{B}$ is the Boltzmann constant. $\varphi_{i}$
($\varphi_{j}$) is the initial (final) ground-state wave function of defect
with the depth $\Delta E_{1}$ ($\Delta E_{2}$), and ${\phi_{0}}$ is the free
carrier state at the band edge. In Eq. (3), ${\varepsilon_{0}}$ is the
permittivity of vacuum, ${\varepsilon_{r}}$ is the relative permittivity, and
$\bf r_{1}$ as well as $\bf r_{2}$ are position variables for electrons. Eq.
(6) describes the interaction between charge carrier and LO phonons in the
Fröhlich mechanismwx moxing5  with the coupling element $\mathcal{M}(q)$,
where ${a_{\bf q}}$ ($a_{-{\bf q}}^{\dagger}$) is annihilation (creation) a LO
phonon with wave vector $\bf q$, $V$ is the volume, $m^{*}$ is the effective
mass of electron (or hole), and $\alpha$ denotes the Fröhlich coupling
strength in this mechanism. The parameter ${\chi_{\lambda}}$ is defined as

${\chi_{\lambda}}=\left\\{\begin{gathered}\frac{\mathcal{S}_{\lambda}}{{p\sinh(\hbar{\omega_{LO}}/2k_{B}T)}}\quad\hfill(\mathcal{S}_{\lambda}<p)\\\
\frac{p}{{\mathcal{S}_{\lambda}\sinh(\hbar{\omega_{LO}}/2k_{B}T)}}\quad\hfill(\mathcal{S}_{\lambda}>p)\end{gathered}\right.,$
(8)

where $\mathcal{S}_{\lambda}$ is Huang-Rhys (HR) factor with the subscript
$\lambda$ = - ($\lambda$ = +) represents the ERT between the same (different)
types of defects, which denotes the average LO phonon numbers around the
trapped carrier, reflecting the strength of the electron-phonon couplingwx
Huang3 ; wx Huang2 ; wx Huang1 .

Following the quantum defect modelwx q1 ; wx q2 ; wx q3 ; wx q4 , the ground-
state wave function $\varphi({r})$ for defect with arbitrary binding energy
can be expressed as

$\varphi(r)=\mathcal{N}(\frac{r}{a^{*}})^{\mu-1}\exp({-\frac{r}{\vartheta
a^{*}}}),$ (9)

with

$\mathcal{N}=\frac{1}{\sqrt{4\pi
a^{*3}(\vartheta/2)^{2\mu+1}\Gamma(2\mu+1)}},$ (10)
$a^{*}=\frac{4\pi\varepsilon\hbar^{2}}{e^{2}m^{*}},$ (11)
$\vartheta=\frac{{{e}}}{{\sqrt{8\pi\varepsilon{a^{*}}\Delta E}}},$ (12)

where $\mathcal{N}$, $a^{*}$, and $\vartheta$ represent normalization
constant, effective Bohr radius, and quantum defect parameter, respectively.
$\Delta E$ denotes the defect depth, the type of defect is reflected by the
parameter $\mu$: $\mu$ = -$\vartheta$ for the donor-like defect, $\mu$ =
+$\vartheta$ for the acceptor-like defect, and $\mu$ = 0 for the neutral
defect. Substituting Eqs. (3), (6), (7) and (9) into Eqs. (2) and (5), the
matrix elements become

$\displaystyle{\left|{{\mathcal{H}_{ij}^{e-e}}}\right|^{2}}$ $\displaystyle=$
$\displaystyle\frac{{64{e^{4}}\mathcal{N}_{1}^{2}\mathcal{N}_{2}^{2}}}{{\varepsilon_{0}^{2}\varepsilon_{r}^{2}a_{0}^{6}}}|\int{\int{\int{d{r_{1}}}}}d{r_{2}}r_{1}^{2}r_{2}^{2}$
(13)
$\displaystyle\times\frac{{\sin(k{r_{1}})\sin(k{r_{2}})}}{{{r_{1}}{r_{2}}{k^{2}}}}{(\frac{{{r_{1}}{r_{2}}}}{{{a^{*2}}}})^{{\mu_{1}}+{\mu_{2}}-2}}$
$\displaystyle\times\exp(-\frac{{{r_{1}}}}{{{\vartheta_{1}}{a^{*}}}}-\frac{{{r_{2}}}}{{{\vartheta_{2}}{a^{*}}}}){dk}|^{2},$

for electron-electron interactionwx Huang2 ; wx ee1 ; wx ee2 ; wx ee3 , and

$\displaystyle|H_{ij}^{e-p}{|^{2}}{\rm{}}$ $\displaystyle=$
$\displaystyle\frac{{16\alpha
a_{0}^{3}\mathcal{N}_{1}^{2}\mathcal{N}_{2}^{2}}}{\pi}{(\hbar{\omega_{LO}})^{2}}{(\frac{\hbar}{{2{m^{*}}{\omega_{LO}}}})^{1/2}}$
(14)
$\displaystyle\times|\int{\int_{0}^{{q_{c}}}{dr{r^{2}}\frac{{\sin(qr)}}{{qr}}{{(\frac{r}{{{a^{*}}}})}^{{\mu_{1}}+{\mu_{2}}-2}}}}$
$\displaystyle\times\exp(-\frac{r}{{{\vartheta_{1}}{a^{*}}}}-\frac{r}{{{\vartheta_{2}}{a^{*}}}})qdq{|^{2}},$

for electron-phonon interactionwx moxing1 ; wx moxing2 . $k$ denotes the
electronic wave vector and $a_{0}$ is the lattice constant.

According to the model proposed by Ridleywx S1 ; wx S2 ; wx S3 , HR factor
between quantum defects can be calculated by

$\mathcal{S}_{\lambda}=\sum\limits_{q}{\frac{{{{\left|\Delta_{\lambda}\right|}^{2}}}}{{2{{(\hbar{\omega_{LO}})}^{2}}}}},$
(15)

with the magnitude parameter

$\Delta_{-}=\mathcal{M}(q)\int{[\varphi}_{i}^{*}(r){e^{iq\cdot
r}}{\varphi_{i}}(r)-\varphi_{j}^{*}(r){e^{iq\cdot
r}}{\varphi_{j}}(r)]{\rm{d}}{\bf{r}},$ (16)

and

$\Delta_{+}=\mathcal{M}(q)\int{[\varphi}_{i}^{*}(r){e^{iq\cdot
r}}{\varphi_{i}}(r)+\varphi_{j}^{*}(r){e^{iq\cdot
r}}{\varphi_{j}}(r)]{\rm{d}}{\bf{r}}.$ (17)

Substituting Eq. (9) into Eqs. (16) and (17), then performing the integral,
one can get

$\displaystyle\Delta_{-}$ $\displaystyle=$
$\displaystyle\mathcal{M}(q)\\{\frac{{\sin(2{\mu_{1}}{\theta_{1}})}}{{2{\mu_{1}}(q{a^{*}}{\vartheta_{1}}/2){{[1+{{(q{a^{*}}{\vartheta_{1}}/2)}^{2}}]}^{{\mu_{1}}}}}}$
(18)
$\displaystyle-\frac{{\sin(2{\mu_{2}}{\theta_{2}})}}{{2{\mu_{2}}(q{a^{*}}{\vartheta_{2}}/2){{[1+{{(q{a^{*}}{\vartheta_{2}}/2)}^{2}}]}^{{\mu_{2}}}}}}\\},$

and

$\displaystyle\Delta_{+}$ $\displaystyle=$
$\displaystyle\mathcal{M}(q)\\{\frac{{\sin(2{\mu_{1}}{\theta_{1}})}}{{2{\mu_{1}}(q{a^{*}}{\vartheta_{1}}/2){{[1+{{(q{a^{*}}{\vartheta_{1}}/2)}^{2}}]}^{{\mu_{1}}}}}}$
(19)
$\displaystyle+\frac{{\sin(2{\mu_{2}}{\theta_{2}})}}{{2{\mu_{2}}(q{a^{*}}{\vartheta_{2}}/2){{[1+{{(q{a^{*}}{\vartheta_{2}}/2)}^{2}}]}^{{\mu_{2}}}}}}\\},$

where ${\theta_{1}}=\arctan(q{a^{*}}{\vartheta_{1}}/2)$ and
${\theta_{2}}=\arctan(q{a^{*}}{\vartheta_{2}}/2)$. Converting the summation of
$q$ into the integral, HR factors can be rewritten as

$\displaystyle\mathcal{S}_{-}$ $\displaystyle=$
$\displaystyle\frac{{\alpha{q_{c}}}}{\pi}{[\frac{\hbar}{{2{m^{*}}{\omega_{LO}}}}]^{1/2}}\int_{0}^{1}{\frac{{{{\sin}^{2}}[{b_{1}}\arctan({c_{1}}x)]}}{{{{({b_{1}}{c_{1}}x)}^{2}}{{[1+{{({c_{1}}x)}^{2}}]}^{{b_{1}}}}}}}dx+\frac{{\alpha{q_{c}}}}{\pi}{[\frac{\hbar}{{2{m^{*}}{\omega_{LO}}}}]^{1/2}}\int_{0}^{1}{\frac{{{{\sin}^{2}}[{b_{2}}\arctan({c_{2}}x)]}}{{{{({b_{2}}{c_{2}}x)}^{2}}{{[1+{{({c_{2}}x)}^{2}}]}^{{b_{2}}}}}}}dx$
(20)
$\displaystyle-\frac{{2\alpha{q_{c}}}}{\pi}\int_{0}^{1}{{{[\frac{\hbar}{{2{m^{*}}{\omega_{LO}}}}]}^{1/2}}}\frac{{\sin[{b_{1}}\arctan({c_{1}}x)]}}{{({b_{1}}{c_{1}}x){{[1+{{({c_{1}}x)}^{2}}]}^{{b_{1}}/2}}}}\times\frac{{\sin[{b_{2}}\arctan({c_{2}}x)]}}{{({b_{2}}{c_{2}}x){{[1+{{({c_{2}}x)}^{2}}]}^{{b_{2}}/2}}}}dx.$

for the transitions between the same types of defects, and

$\displaystyle\mathcal{S}_{+}$ $\displaystyle=$
$\displaystyle\frac{{\alpha{q_{c}}}}{\pi}{[\frac{\hbar}{{2{m^{*}}{\omega_{LO}}}}]^{1/2}}\int_{0}^{1}{\frac{{{{\sin}^{2}}[{b_{1}}\arctan({c_{1}}x)]}}{{{{({b_{1}}{c_{1}}x)}^{2}}{{[1+{{({c_{1}}x)}^{2}}]}^{{b_{1}}}}}}}dx+\frac{{\alpha{q_{c}}}}{\pi}{[\frac{\hbar}{{2{m^{*}}{\omega_{LO}}}}]^{1/2}}\int_{0}^{1}{\frac{{{{\sin}^{2}}[{b_{2}}\arctan({c_{2}}x)]}}{{{{({b_{2}}{c_{2}}x)}^{2}}{{[1+{{({c_{2}}x)}^{2}}]}^{{b_{2}}}}}}}dx$
(21)
$\displaystyle+\frac{{2\alpha{q_{c}}}}{\pi}\int_{0}^{1}{{{[\frac{\hbar}{{2{m^{*}}{\omega_{LO}}}}]}^{1/2}}}\frac{{\sin[{b_{1}}\arctan({c_{1}}x)]}}{{({b_{1}}{c_{1}}x){{[1+{{({c_{1}}x)}^{2}}]}^{{b_{1}}/2}}}}\times\frac{{\sin[{b_{2}}\arctan({c_{2}}x)]}}{{({b_{2}}{c_{2}}x){{[1+{{({c_{2}}x)}^{2}}]}^{{b_{2}}/2}}}}dx.$

for the transitions between different types of defects.
${q_{c}}=\sqrt[3]{{6{\pi^{2}}}}/{a_{0}}$ is the radius of sphere with
Brillouin zone volume. ${b_{1}}=2{\mu_{1}}$, ${b_{2}}=2{\mu_{2}}$,
${c_{1}}={q_{c}}{a^{*}}{\vartheta_{1}}/2$,
${c_{2}}={q_{c}}{a^{*}}{\vartheta_{2}}/2$, and $q=q_{c}x$. In this paper, we
choose one of typical MHPs CH3NH3PbI3 (MAPbI3) as an example to discuss these
transition processes. The values of parameters for the MAPbI3 are given in
Table I.

Figure 3:  Lifetimes of the non-radiative multiphonon energy resonance
transfer between donor-like defects as a function of the emitted phonon number
$p$ in the mechanisms of the electron-electron interaction $\tau_{e-e}$ (a)
and the electron-phonon interaction $\tau_{e-p}$ (b) for the different depths
of the initial defect $\Delta E_{1}$ at $T$ = 300 K, where the final defect
depth $\Delta E_{2}$ satisfies the relation of $\Delta E_{2}=\Delta
E_{1}+p\hbar{\omega_{LO}}$. The temperature dependences of lifetime for
different phonons emission between donor-like defects in the mechanisms of the
electron-electron interaction (c) and the electron-phonon interaction (d), in
which a fixed initial defect depth $\Delta E_{1}=0.2$ eV is selected.

##  III results and discussion

HR factor is a key parameter determining the lifetime of ERT, which was
originally proposed by Huang for F-centres in ionic crystals in 1950wx Huang1
and denotes the average phonon numbers around F-centres, reflecting the
magnitude of lattice distortion arising from the strong coupling between
electronic states and lattice vibration (phonons). The magnitude could be
expressed by the “phonon cloud” and its energy is described by
$\mathcal{S}_{\lambda}\hbar{\omega_{LO}}$, namely the lattice relaxation
energy. The dependences of HR factors ($\mathcal{S}_{-}$) on the depth
difference between initial ($\Delta E_{1}$) and final ($\Delta E_{2}$) defect
levels are shown in Figs. 2 (a) and (b) for the donor-donor and acceptor-
acceptor transitions, respectively. One can see that $\mathcal{S}_{-}$
increases with increasing the depth separation between $\Delta E_{1}$ and
$\Delta E_{2}$. However, its value only steps up to 0.25 and 1.25 for the
donor-like and acceptor-like defects, respectively. Although a single quantum
defect exhibits a significant lattice relaxation effect (namely, the big value
of HR factor)wx moxing1 , the relative lattice relaxation effect between two
defect levels in the same type is not strong enough, which is, in fact, not
beneficial to the non-radiative processes between two defect levels. On the
contrary, HR factors ($\mathcal{S}_{+}$) for donor-acceptor transitions can
reach in the range of $26\sim 32$, depending on the depth separation between
donor-like and acceptor-like defect shown in Fig. 2 (c). This strong lattice
relaxation effect, stemming from the sum of the contributions from the
electron and hole overlapping with each other, will promote defect-assisted
non-radiative transitions.

Figure 4:  Lifetimes of the non-radiative multiphonon energy resonance
transfer between donor- and acceptor-like defects as a function of the emitted
phonon number $p$ in the mechanisms of the electron-electron interaction (a)
and the electron-phonon interaction (b) for the different depths of the
initial defect $\Delta E_{1}$ at $T$ = 300 K. The final defect depth $\Delta
E_{2}$ satisfies the relation of
$\Delta{E_{2}}=\left|{{E_{C}}-{E_{V}}}\right|-\Delta{E_{1}}-p\hbar{\omega_{LO}}$,
where $\left|{{E_{C}}-{E_{V}}}\right|$ is the bandgap.

We first analyze the dependences of non-radiative multiphonon ERT between
donor-like defects on the emitted phonon number $p$ and the initial defect
depth $\Delta E_{1}$, where the final defect depth satisfies the relation of
$\Delta E_{2}=\Delta E_{1}+p\hbar{\omega_{LO}}$. The multiphonon ERT induced
by electron-electron and electron-phonon interactions are shown in Figs. 3(a)
and (b), respectively. As can be seen that the lifetime ($\tau_{e-e}$ or
$\tau_{e-ph}$ ) has an obvious increase of more than one order of magnitude
when $p$ increases by one, which stems from the weak lattice relaxation effect
between the same types of defects as given in Fig. 2(a). With the increasing
of the initial defect depth $\Delta E_{1}$, these processes of multiphonon ERT
are also significantly suppressed. For example, the lifetime can maintain in
the range of hundreds of picoseconds with the successive emitting five phonons
at $\Delta E_{1}=0.1$ eV as shown in Fig. 3(a) induced by electron-electron
interaction. But, at $\Delta E_{1}=0.6$ eV, the process of three phonons
already reaches tens of nanoseconds, which indicates that charge carries can
transfer effectively by multiphonon processes between the neighbouring shallow
defect levels, causing the decrease of the carrier concentration seriously.
Moreover, from the comparisons between Figs. 3(a) and (b), one can see that
the electron transfer processes induced by the electron-electron interaction
are much more faster (about three orders of magnitude) than the same processes
induced by the electron-phonon interaction, which reveals that the electron-
electron interaction will dominate the non-radiative ERT processes among
quantum defects. The temperature dependence of lifetimes of these processes
are plotted in Figs. 3(c) and (d) for the electron-electron and electron-
phonon interactions, respectively. One can see that lifetimes of these
multiphonon processes become shorter with the increasing of the temperature,
which means that temperature increasing makes more phonons are excited,
resulting in carriers are more likely to be captured by deeper defects. The
similar results are also obtained for acceptor-like defects and neutral
defects shown in the supplemental materials.

Table 2:  Huang-Rhys factor $\mathcal{S}_{+}$ are calculated by Eq. (21) for the different depths of the final acceptor-like defect $\Delta E_{2}$. The depth of the initial donor-like defect $\Delta E_{1}$=0.4 eV is fixed. Final defect depth ($\Delta E_{2}$) |  0.7875 eV |  0.705 eV |  0.54 eV |  0.4575 eV
---|---|---|---|---
Phonon number ($p$) |  25 |  30 |  40 |  45
HR factor ($\mathcal{S}_{+}$) |  31.7 |  31.2 |  30 |  29.2
Figure 5:  The temperature dependence of the lifetimes of multiphonon
processes between donor- and acceptor-like defects for the different phonons
$p$ emission in the mechanisms of the electron-electron interaction (a) and
the electron-phonon interaction (b), where a fixed depth of the initial defect
$\Delta E_{1}=0.4$ eV is selected. (c) The parameter $\chi_{+}$ as a function
of the temperature $T$ for the different phonon number $p$.

Figs. 4(a) and (b) show the non-radiative ERT processes between donor- and
acceptor-like defects for the mechanisms of electron-electron interaction and
electron-phonon interaction, respectively, in which the relationship between
the initial and final defect depths satisfies
$\Delta{E_{2}}=\left|{{E_{C}}-{E_{V}}}\right|-\Delta{E_{1}}-p\hbar{\omega_{LO}}$,
where $\left|{{E_{C}}-{E_{V}}}\right|$ is the band gap. It can be seen that
these non-radiative multiphonon processes are very fast for different initial
(donor-like) defect depths $\Delta E_{1}=$ 0.4 eV, 0.5 eV, and 0.6 eV. Even at
the large emitted phonon number $p=50$, the lifetime varies just only in one
order of magnitude scale, which arises from the strong lattice relaxation
effect (namely the large HR factors) as shown in Fig. 2 (c). These ultrafast
processes will result in a dramatic reduction of free carriers concentration
in a short time, which reveal that the charge carriers can be transferred
effectively via multiphonon processes between donor- and acceptor-like defects
at any depth. Recently, Merdasa $et$ $al.$ observed large fluctuations of
luminescence intensity, which was often referred to as a blinking phenomenon
in MAPbI3 by luminescence super-resolution imaging and spectroscopywx r6 .
They ascribed this phenomenon to the ultrafast non-radiative recombination via
the “supertraps” mechanism (donor-acceptor pairs). Meanwhile, the non-
radiative process via these “supertraps” at low defect concentrations was much
more efficient than via other single defect states at higher concentrations.
Therefore, these rapid non-radiative multiphonon transitions from donor- to
acceptor-like defects obtained in the present model could provide a possible
explanation for this mechanism of “supertraps”.

The temperature dependences of multiphonon transitions from donor- to
acceptor-like defects are shown in Figs. 5(a) and (b) for the mechanisms of
electron-electron interaction and electron-phonon interaction, respectively.
One can find that the variational trends of the lifetime with temperature
depend on the comparison of HR factor $\mathcal{S}_{+}$ with $p$ (the
relationship between $\mathcal{S}_{+}$ and $p$ is shown in Table II). When
$\mathcal{S}_{+}$ is bigger than $p$, the lifetime increases with temperature.
This is because the relaxation energy ($\mathcal{S}_{+}\hbar{\omega_{LO}}$)
are inclined to relax into the environment directly, but the enhancement of
lattice vibration by the temperature rising disturbs the direct relaxation
processes. When $\mathcal{S}_{+}$ is smaller than $p$, the relaxation
processes depend on the temperature exciting more phonons to compensate the
difference ($p-\mathcal{S}_{+}$), and thus results in multiphonon processes
become faster with temperature. In addition, all the curves vary hardly in low
temperature region ($T$$<$25 K), which is very consistent with the temperature
turning point of $\chi_{+}$ as shown in Fig. 5(c). This suggests that
$\chi_{+}$ is also a key parameter to determine the temperature dependence of
these non-radiative multiphonon processes in this model. Finally, we must
emphasize that MAPbI3 undergoes two structural phase transitions with
temperaturewx phase1 ; wx phase2 ; wx phase3 , which has the significant
influence on the strength of the electron-phonon interaction and phonon
energy. These effects are not taken into account in the present paper.

##  IV conclusion

In conclusion, we theoretically study the non-radiative multiphonon processes
of energy resonance transfer among quantum defects for the induced mechanisms
of the electron-electron interaction and electron-phonon interactions,
respectively. We find that (1) charge carrier can transfer by non-radiation
multiphonon processes between the neighboring levels of the same type shallow
defects, but it will be suppressed with the increasing of the defect depth;
(2) non-radiation multiphonon processes between donor- and acceptor-like
defects are very fast and independence of the defect depth, which provides a
possible explanation for the “supertraps”of the charge carriers in recent
experiment; (3) the comparison of Huang-Rhys factor $\mathcal{S_{+}}$ with the
emitted phonon number $p$ plays an important role in determining the
variational trends of temperature dependence of these multiphonon processes.

This work was supported by National Natural Science Foundation of China (Grand
No. 11674241).

## References

  * (1) M. A. Green, A. Ho-Baillie, and H. J. Snaith, Nat. Photonics 8, 506 (2014).
  * (2) L. Protesescu, S. Yakunin, M. I. Bodnarchuk, F. Krieg, R. Caputo, C. H. Hendon, R. X. Yang, A. Walsh, and M. V. Kovalenko, Nano Lett. 15, 3692 (2015).
  * (3) Y.-H. Kim, H. Cho, and T.-W. Lee, Proc. Natl. Acad. Sci. U. S. A. 113, 11694 (2016).
  * (4) H. Huang, M. I. Bodnarchuk, S. V. Kershaw, M. V. Kovalenko, and A. L. Rogach, ACS Energy Lett. 2, 2071 (2017).
  * (5) S. Singh, Laxmi, and D. Kabra, J. Phys. D: Appl. Phys. 53, 503003 (2020).
  * (6) S. D. Stranks, ACS Energy Lett. 2, 1515 (2017).
  * (7) T. Kirchartz and U. Rau, Adv. Energy Mater. 8, 1703385 (2018).
  * (8) J.-S. Park, S. Kim, Z. Xie, and A. Walsh, Nat. Rev. Mater. 3, 194 (2018).
  * (9) D. Luo, R. Su, W. Zhang, Q. Gong, and R. Zhu, Nat. Rev. Mater. 5, 44 (2020).
  * (10) H. Jin, E. Debroye, M. Keshavarz, I. G. Scheblykin, M. B. J. Roeffaers, J. Hofkens, and J. A. Steele, Mater. Horiz. 7, 397 (2020).
  * (11) W. Tress, Adv. Energy Mater. 7, 1602358 (2017).
  * (12) C. M. Wolff, P. Caprioglio, M. Stolterfoht, and D. Neher, Adv. Mater 31, 1902762 (2019).
  * (13) D. W. deQuilettes, S. M. Vorpahl, S. D. Stranks, H. Nagaoka, G. E. Eperon, M. E. Ziffer, H. J. Snaith, and D. S. Ginger, Science 348, 683 (2015).
  * (14) Y. Yamada, T. Nakamura, M. Endo, A. Wakamiya, and Y. Kanemitsu, J. Am. Chem. Soc. 136, 11610 (2014).
  * (15) G.-J. A. H. Wetzelaer, M. Scheepers, A. M. Sempere, C. Momblona, J. Ávila, and H. J. Bolink, Adv. Mater. 27, 1837-1841 (2015).
  * (16) W.-J. Yin, T. Shi, and Y. Yan, Appl. Phys. Lett. 104, 063903 (2014).
  * (17) D. W. deQuilettes, K. Frohna, D. Emin, T. Kirchartz, V. Bulovic, D. S. Ginger, and S. D. Stranks, Chem. Rev. 119, 11007-11019 (2019).
  * (18) Y. Rakita, I. Lubomirsky, and D. Cahen, Mater. Horiz. 6, 1297-1305 (2019).
  * (19) O. D. Miller, E. Yablonovitch, S. R. Kurtz, IEEE J. Photovolt. 2, 303-311 (2012).
  * (20) W. Tress, N. Marinova, O. Inganäs, M. K. Nazeeruddin, S. M. Zakeeruddin, M. Graetzel, Adv. Energy Mater. 5, 1400812 (2015).
  * (21) Z.-Y. Zhang, H.-Y. Wang, Y.-X. Zhang, Y.-W. Hao, C. Sun, Y. Zhang, B.-R. Gao, Q.-D. Chen, and H.-B. Sun, Sci Rep 6, 27286 (2016).
  * (22) S. D. Stranks, ACS Energy Lett. 2, 1515-1525 (2017).
  * (23) T. Markvart, J. Phys. C: Solid State Phys. 14, 895 (1981).
  * (24) T. Markvart, J. Phys. C: Solid State Phys. 14, 435 (1981).
  * (25) T. Markvart, Multiphonon recombination. In $recombination$ in $Semiconductors$, Landsberg, P. T., Ed.; Cambridge University Press:Cambridge, U.K., 2003; pp 467-468.
  * (26) T. Kirchartz, T. Markvart, U. Rau, and D. A. Egger, J. Phys. Chem. Lett. 9, 939 (2018).
  * (27) A. D. Wright, C. Verdi, R. L. Milot, G. E. Eperon, M. A. P$\acute{e}$rez-Osorio, H. J. Snaith, F. Giustino, M. B. Johnston, and L. M. Herz, Nat. Commun 7, 11755 (2016).
  * (28) D. Zhao, H. Hu, R. Haselsberger, R. A. Marcus, M. E. Michel-Beyerle, Y. M. Lam, J.-X. Zhu, C. La-o-vorakiat, M. C. Beard, and E. E. M. Chia, ACS Nano 13, 8826 (2019).
  * (29) J. T. Devreese, arXiv:1012.4576 (2018).
  * (30) K. Huang and A. Rhys, Proc. R. Soc. A 204, 406 (1950).
  * (31) K. Huang, Contemp. Phys. 22, 599 (1981).
  * (32) Z.-W. Wang, Y. Xiao, J.-P. Deng, Y. Cui, and Z.-Q, Li, Phys. Rev. B 100, 125308 (2019).
  * (33) H. B. Bebb and R. A. Chapman, J. Phys. Chem. Solids 28, 2087 (1967).
  * (34) H. B. Bebb, Phys. Rev. 185, 1116 (1969).
  * (35) S. T. Pantelides, Rev. Mod. Phys. 50, 797 (1978).
  * (36) D. Swiatla and W. M. Bartczak, Phys. Rev. B 43, 6776 (1990).
  * (37) U. Bockelmann and T. Egeler, Phys. Rev. B 46, 15574 (1992).
  * (38) E. Martin, C. Deleru, G. Allan, and M. Lannoo, Phys. Rev. B 50, 18258 (1994).
  * (39) A. Tomadin, D. Brida, G. Cerullo, A. C. Ferrari, and M. Polini, Phys. Rev. B 88, 035430 (2013).
  * (40) B. K. Ridley, Solid-State Electron. 21, 1319 (1978).
  * (41) B. K. Ridley, J. Phys. C: Solid State Phys. 11, 2323 (1978).
  * (42) D. Światla and W. M. Bartczak, Phys. Rev. B 43, 6776 (1991).
  * (43) B. Das, I. Aguilera, U. Rau, and T. Kirchartz, Phys. Rev. Materials 4, 024602 (2020).
  * (44) A. Poglitsch and D. Weber, J. Chem. Phys. 87, 6373 (1987).
  * (45) A. Merdasa, Y. Tian, R. Camacho, A. Dobrovolsky, E. Debroye, E. L. Unger, J. Hofkens, V. Sundström, and I. G. Scheblykin, ACS Nano 11, 5391 (2017).
  * (46) P. S. Whitfield, N. Herron, W. E. Guise, K. Page, Y. Q. Cheng, I. Milas, and M. K. Crawford, Sci Rep 6, 35685 (2016).
  * (47) L. Liu, R. Zhao, C. Xiao, F. Zhang, F. Pevere, K. Shi, H. Huang, H. Zhong, and I. Sychugov, J. Phys. Chem. Lett. 10, 5451 (2019).
  * (48) M. Keshavarz, M. Ottesen, S. Wiedmann, M. Wharmby, R. Küchler, H. Yuan, E. Debroye, J. A. Steele, J. Martens, N. E. Hussey, M. Bremholm, M. B. J. Roeffaers, and J. Hofkens, Adv. Mater. 31, 1900521 (2019).