# PaCoST: Paired Confidence Significance Testing for Benchmark Contamination
Detection in Large Language Models

Huixuan Zhang∗ Yun Lin∗ Xiaojun Wan†
Wangxuan Institute of Computer Technology, Peking University
pkuleozhang@gmail.com,linyun<EMAIL_ADDRESS>
###### Abstract

Large language models (LLMs) are known to be trained on vast amounts of data,
which may unintentionally or intentionally include data from commonly used
benchmarks. This inclusion can lead to cheatingly high scores on model
leaderboards, yet result in disappointing performance in real-world
applications. To address this benchmark contamination problem, we first
propose a set of requirements that practical contamination detection methods
should follow. Following these proposed requirements, we introduce PaCoST, a
Paired Confidence Significance Testing to effectively detect benchmark
contamination in LLMs. Our method constructs a counterpart for each piece of
data with the same distribution, and performs statistical analysis of the
corresponding confidence to test whether the model is significantly more
confident under the original benchmark. We validate the effectiveness of
PaCoST and apply it on popular open-source models and benchmarks. We find that
almost all models and benchmarks we tested are suspected contaminated more or
less. We finally call for new LLM evaluation methods. 111 Our code will be
released to the community.

PaCoST: Paired Confidence Significance Testing for Benchmark Contamination
Detection in Large Language Models

Huixuan Zhang∗ Yun Lin∗ Xiaojun Wan† Wangxuan Institute of Computer
Technology, Peking University
pkuleozhang@gmail.com,linyun<EMAIL_ADDRESS>

††∗ Both authors contributed equally to this research.††† Corresponding
author.

##  1 Introduction

Large Language Models (LLMs) have brought about a paradigm shift in the domain
of natural language processing, yielding notable enhancements across various
evaluation benchmarks Wang et al. (2019) and demonstrating proficiency in
professional examinationsOpenAI (2023). These advancements primarily stem from
extensive training on vast and diverse datasets sourced from multiple origins.
However, the substantial volume of data has given rise to significant concerns
regarding benchmark contamination, where benchmarks for LLM evaluation are
inadvertently or deliberately included in model training. This contamination
presents considerable obstacles in accurately gauging the capabilities of
LLMs.

While efforts are being made to address this issue by removing benchmarks from
training datasets and conducting contamination studies, these endeavors face
numerous limitations Brown et al. (2020a); Zhang et al. (2024); Wei et al.
(2022); Chowdhery et al. (2022). These limitations include narrow focus on
specific benchmarks and reliance on the trustworthiness of vendors. Moreover,
the competitive dynamics within the field, coupled with copyright
considerations, have resulted in recent model releases lacking accompanying
contamination studies OpenAI (2023). Hence, there is an urgent necessity for
independent methods to audit LLMs for the presence of benchmark datasets,
eliminating the dependence on model providers’ cooperation.

Simultaneously, there has been a growing interest in heuristic membership
inference algorithms designed to reverse-engineer aspects of the training
dataset Carlini et al. (2021a); Mattern et al. (2023), thereby providing
insights into potential test set contamination Sainz et al. (2023a); Golchin
and Surdeanu (2023b). Despite their promise, these heuristic approaches often
lack definitive proof of contamination and tend to rely on assumptions that
may be too stringent. Moreover, the majority of these methods concentrate less
on detecting benchmark contamination. As elaborated in Section 3.1, inherent
challenges, such as the need for lengthy trained segments and the necessity of
establishing thresholds, impede the adaptation of previous methods for
detecting benchmark contamination.

In this study, we introduce a novel approach named PaCoST (Paired Confidence
Significance Testing) designed for the detection of benchmark contamination in
open-source LLMs. Our method entails a three-step statistical analysis,
capable of identifying benchmarks within the model’s training data.
Specifically, our approach involves constructing counterparts for each data
instance with similar distribution, followed by statistical analysis of
corresponding confidence scores to ascertain whether the model exhibits
significantly higher confidence when presented with original benchmarks. We
operate under the assumption that the model tends to demonstrate greater
confidence when responding to questions it has been trained on. To validate
our method rigorously, we conduct a series of controlled experiments.

Subsequently, we employ PaCoST across a diverse array of publicly accessible
LLMs, scrutinizing various benchmarks to reveal contamination outcomes. Our
experimental observations indicate that, across the board, there are
suspicions of contamination to varying degrees in both models and benchmarks.
Consequently, we advocate for the adoption of a benchmark-free evaluation
approach as a means to mitigate this contamination issue.

Our contributions can be summarized as follows:

  * •

We propose several properties which a good benchmark contamination detection
method should satisfy.

  * •

We introduce a simple yet effective method PaCoST to detect benchmark
contamination in LLMs and validate its effectiveness and stability.

  * •

We conduct experiments on popular open-source LLMs and benchmarks and find
suspected contamination on almost all tested models and benchmarks.

##  2 Related Works

###  2.1 Data Contamination Detection

The issue of data contamination in large language models has been increasingly
recognized as a significant concern (Sainz et al., 2023a). Many LLM providers
use string-matching to report contamination, such as GPT-2 Radford et al.
(2019), GPT-3 Brown et al. (2020b), PaLM Chowdhery et al. (2023), GPT-4 OpenAI
(2023), and Llama 2 Touvron et al. (2023). However, in most cases, the model’s
training data is not publicly available, necessitating alternative detection
methods.

Several methods have been developed to detect data contamination in LLMs. Nasr
et al. (2023) and Sainz et al. (2023b) explore the regeneration of initial
dataset instances. Golchin and Surdeanu (2023b) introduces guided prompting to
replicated trained data. Golchin and Surdeanu (2023a) develops a Data
Contamination Quiz (DCQ) framework.

Beyond prompt-based methods, there are also methods based on likelihood such
as the Min-K% Prob Shi et al. (2024), Oren et al. (2023) and Li (2023).
Additionally, methodologies like CDD and TED (Dong et al., 2024) focus on the
LLM’s output distribution. But these methods do not pay enough attention to
benchmark contamination detection.

Membership Inference Attack (MIA) is closely related to data contamination,
aiming to identify whether a given sample is in a model’s training data
(Shokri et al., 2017; Yeom et al., 2018). These attacks pose significant
privacy risks and can lead to severe breaches Carlini et al. (2021b); Gupta et
al. (2022); Cummings et al. (2023). MIA is crucial for assessing privacy
vulnerabilities and validating privacy-preserving measures in machine learning
models Jayaraman and Evans (2019); Jagielski et al. (2020); Nasr et al.
(2023). Initially applied to tabular and computer vision data, MIA has
recently been extended to language-based tasks Song and Shmatikov (2019);
Shejwalkar et al. (2021); Mahloujifar et al. (2021); Mireshghallah et al.
(2022).

Method | TDA Free | CT Free | TDL Free | SP | T Free
---|---|---|---|---|---
String-match (OpenAI, 2023) | ✗ | ✓ | ✓ | ✓ | ✓
Min-k% Prob (Shi et al., 2024) | ✓ | ✗ | ✗ | ✓ | ✗
Guided-Prompting (Golchin and Surdeanu, 2023b) | ✓ | ✗ | ✗ | ✗ | ✓
Sharded-Likelihood (Oren et al., 2023) | ✓ | ✗ | ✗ | ✓ | ✗
CDD (Dong et al., 2024) | ✓ | ✗ | ✗ | ✗ | ✗
DCQ (Golchin and Surdeanu, 2023a) | ✓ | ✓ | ✓ | ✗ | ✓
PaCoST (ours) | ✓ | ✓ | ✓ | ✓ | ✓
Table 1: Comparison of existing methods and PaCoST. ✓ means the method
satisfies the corresponding property and ✗ refers to methods not satisfying
the corresponding property. The name of properties are abbreviated for
presentation and their full contents can be found in Section 3.2.

###  2.2 Confidence Estimation

Estimating the confidence of a model in its output is a critical challenge in
the research of LLMs. Kuhn et al. (2023) aggregates probabilities of
semantically equivalent answers to determine confidence. Other methods include
directly querying the model for its confidence Lin et al. (2022); Tian et al.
(2023) and calculating self-consistency scores Wang et al. (2022); Lin et al.
(2023). Some techniques for confidence calibration involve modifying prompts
and paraphrasing instructions to fine-tune the probability distribution (Zhao
et al., 2023; Jiang et al., 2023b), or using the probability that the model
agrees with its own answers, such as in P(True) (Kadavath et al., 2022).
Combined approaches further enhance calibration accuracy Xiong et al. (2023);
Chen and Mueller (2023).

##  3 Problem Formulation

###  3.1 Benchmark Contamination

In this study, we focus on detecting benchmark contamination. The problem is
formulated as: consider a benchmark $D=\\{(x_{1},y_{1}),...,(x_{n},y_{n})\\}$,
where $x_{i}$ denotes an instruction and $y_{i}$ represents the ground truth
answer. We define benchmark contamination as the model has been trained to
maximize $\mathcal{P}(y_{i}|x_{i})$ (or to minimize
$-\log\mathcal{P}(y_{i}|x_{i})$).

There are two contamination types that align with this objective. For a given
data instance $(x,y)$, the first contamination type performs next-token
prediction on both the instruction $x$ and the answer $y$, which aims at
minimizing:

$\displaystyle-\log\mathcal{P}(x,y)$
$\displaystyle=-\log\mathcal{P}(y|x)\mathcal{P}(x)$
$\displaystyle=-(\log\mathcal{P}(y|x)+\log\mathcal{P}(x))$

The second contamination type only performs next-token prediction on the
answer $y$, which aims at minimizing $-\log\mathcal{P}(y|x)$. The only
difference between the two contamination types lies in whether
$-\log\mathcal{P}(x)$ is part of the optimizing objective.

###  3.2 Detection Requirements

Building upon the formulation outlined earlier and taking into account the
features of existing methodologies for detecting data contamination, we
propose several key criteria that a robust benchmark contamination detection
method should satisfy.

#### I. Training Data Access Free (TDA Free)

While String-Match might offer high accuracy in detecting data contamination,
it is frequently impractical due to LLM providers’ reluctance to disclose
training datasets. Even if training datasets were accessible, the sheer volume
of data makes pinpointing specific instances nearly impossible. Hence,
reliance on access to original training data for contamination detection is
neither feasible nor practical. Effective benchmark contamination detection
methods must be engineered to operate independently of training data access.

#### II. Contamination Type Free (CT Free)

Many conventional contamination detection methods primarily target the first
type of contamination, where both the instruction and answer parts are
trained. This focus is reasonable for detecting contamination in unlabeled
data. However, benchmark contamination can also manifest in the second type,
where only the answer part undergoes training, rendering many existing methods
unsuitable for addressing this issue. For example, techniques like Min-k% Prob
(Shi et al., 2024), which entails computing the minimum k% probabilities of
the entire input, may fail to function accurately if the instruction part
remains untrained. Hence, an effective detection method should not be
constrained by contamination type.

#### III. Training Data Length Free (TDL Free)

Building on the preceding discussion, since most data contamination detection
methods focus on the first type of contamination, they naturally presume
relatively lengthy trained parts. However, benchmark contamination can also
occur with only a very short answer part $y$ trained (e.g., merely an option
or a word). This renders assumptions about training length invalid, making
methods reliant on such assumptions ineffectual. Still taking Min-k% Prob (Shi
et al., 2024) as an example, if we solely compute the minimum k% probabilities
on the response part, it will introduce excessive noise due to the brevity of
the response part. Hence, a robust benchmark contamination detection method
should not be constrained by training length. This will be further discussed
in Appendix B.

#### IV. Stable Performance (SP)

Certain methods exhibit sensitivity to prompts or settings, necessitating
specific prompts for proper functionality. Guided-Prompting (Golchin and
Surdeanu, 2023b) mandates knowledge of the dataset’s name and split to
construct the guided prompt, which may not always be attainable. Moreover, the
disparity between general and guided prompts, even without considering dataset
metadata, casts doubt on the method’s stability. Similarly, DCQ (Golchin and
Surdeanu, 2023a) mandates the model to choose from five options, and altering
the order of options yields disparate results, rendering its detection
outcomes meaningless. Therefore, a robust detection method should yield stable
results despite reasonable changes in settings.

#### V. Threshold Free (T Free)

Some methods necessitate the selection of a threshold for detection, such as
Min-k% Prob (Shi et al., 2024). However, datasets and models exhibit varying
distributions, rendering a universal threshold impractical. While some
threshold-sensitive methods resort to reporting Area Under the Curve (AUC) for
quantitative comparison to circumvent this issue, in real-world scenarios,
employing a specific threshold for detection is unavoidable. Therefore, we
contend that a threshold-based method should provide a fixed threshold and
demonstrate its effectiveness across all datasets rather than relying on AUC.
A superior detection method should not entail flexible thresholds; all
hyperparameters should be predefined.

We examine the most popular data contamination detection methods and compare
them in Table 1. As evident, all methods, except our proposed one, fail to
satisfy all properties. This observation underscores the advantages of our
method.

Figure 1: Overview of our method. $x_{i}$ represents a question, $y_{i}$
represents its corresponding ground truth answer, $x_{i}^{{}^{\prime}}$
represents a rephrased question and $a_{i},a_{i}^{{}^{\prime}}$ represent
model responses to original and rephrased question correspondingly.

##  4 Method

We introduce PaCoST, a novel benchmark contamination detection method that
emphasizes the distinction between contaminated and clean data without relying
on thresholds. Our approach leverages the disparity in model behavior between
original and rephrased instances, focusing on confidence rather than
traditional performance metrics like accuracy (Yang et al., 2023). By
conducting statistical analysis on confidence, we can robustly identify
contamination. PaCoST comprises three key steps: rephrasing preparation,
confidence estimation, and significance testing. Through this method, we
provide a clear and unique approach to detecting benchmark contamination in
models.

###  4.1 Rephrase Preparation

Our key idea involves comparing confidence between original and rephrased
instances. We opt for rephrasing for several reasons. First, to ensure a fair
comparison, the trained and untrained data should share similar distributions
and levels of difficulty. Otherwise, comparing confidence would be
meaningless. Creating questions with the same distribution and difficulty but
different meanings is challenging. Second, rephrasing is a fundamental
capability of most common LLMs, making it straightforward to implement.

Given an instance $(x,y)$, we use a model $M_{p}$ to rephrase $x$ into
$x^{\prime}=M_{p}(x)$ while $y$ remain unchanged. We select Llama2-Chat-7B
(Touvron et al., 2023) as the rephrase model for all the tested models (The
rephrase prompt is provided in Appendix D). To validate the quality of the
rephrasing, we employ both BERT-Score (Zhang* et al., 2020) and human
annotation. Additionally, we compare the performance of different models for
rephrasing and demonstrate that using various paraphrasing models does not
impact performance, provided they are sufficiently powerful. Further details
can be found in Appendix C.

###  4.2 Confidence Estimation

There are various ways to estimate a model’s confidence in its answers, as
previously discussed. In this study, we select the method P(True) (Kadavath et
al., 2022) for confidence estimation.

We briefly introduce this method. Consider an instance $(x,y)$, where $x$ is
an instruction and $y$ is the ground truth answer. For an LLM $M$ and its
corresponding output $M(x)$, P(True) queries the model $M$ whether $M(x)$ is a
correct answer to $x$. Denote the output probability distribution of querying
as $\mathcal{P}(\cdot|x,M(x),M)$, the confidence can be then denoted as
$\mathcal{P}(True|x,M(x),M)$ where True represents model $M$ supporting
$M(x)$. According to our setting and prompt, we are actually calculating
$\mathcal{P}(Yes|x,M(x),M)$.

We opt for using P(True) for confidence estimation for several reasons. First,
using probability distribution of the original output ($\mathcal{P}(M(x)|x,M)$
to estimate confidence often leads to overconfidence issues, resulting in
unnaturally high confidence scores (Xiong et al., 2023). This problem also
partly explains why methods like Min-k% Prob are ineffective on relatively
short training segments. We will further explore this observation in Appendix
B.

Second, Verbalized confidence estimation methods, which involve directly
querying the model to provide a confidence score, often yield discrete
confidence values. This makes them unsuitable for our purposes. Other
confidence estimation methods are generally either inappropriate or overly
complex. Therefore, we ultimately choose P(True) for its simplicity and
effectiveness. Details of our prompt and an example can be found in Appendix
D.

###  4.3 Significance Testing

Consider a benchmark $D$ $=$ $\\{(x_{1},y_{1}),...,$ $(x_{n},y_{n})\\}$ and
its rephrased benchmark
$D^{{}^{\prime}}=\\{(x_{1}^{{}^{\prime}},y_{1}),...,(x_{n}^{{}^{\prime}},y_{n})\\}$
we have calculated the paired confidence set
$\\{(c_{1},c_{1}^{{}^{\prime}}),...,(c_{n},c_{n}^{{}^{\prime}})\\}$, where

$c_{i}=\mathcal{P}(Yes|x_{i},M(x_{i}),M)$

and

$c_{i}^{{}^{\prime}}=\mathcal{P}(Yes|x_{i}^{{}^{\prime}},M(x_{i}^{{}^{\prime}}),M)$

We use Paired Samples T-test to perform statistical analysis. Denote
$d_{i}=c_{i}-c_{i}^{{}^{\prime}}$, assuming
$d_{i}\sim\mathcal{N}(\mu,\sigma^{2})$, we would like to test whether $\mu>0$.
Then the null hypothesis $H_{0}$ and the alternative hypothesis can be denoted
as

$H_{0}:\mu\leq 0\longleftrightarrow H_{1}:\mu>0$

We have:

$\bar{d}=\dfrac{1}{n}\sum_{i=1}^{n}(c_{i}-c_{i}^{{}^{\prime}})$

and

$s_{d}=\sqrt{\dfrac{1}{n-1}\sum_{i=1}^{n}(d_{i}-\bar{d})^{2}}$

the corresponding t-value is

$t=\dfrac{\bar{d}}{\frac{s_{d}}{\sqrt{n}}}$

.

After calculating this t-value, we can calculate a probability $p$ (following
the setting of T-test), which represents the probability of mis-rejecting the
null hypothesis. If $p<0.05$, we can confidently reject null hypothesis and
choose the alternative hypothesis, which means the model is statistically
significantly more confident when answering the original questions and this
provides evidence for potential contamination.

In short, if the calculated $p<0.05$, we say the model $M$ is contaminated on
benchmark $D$, otherwise we say there is no statistically significant evidence
of contamination.

The whole process of our method is shown in Algorithm 1.

Algorithm 1 PaCoST

1: Input benchmark $D=\\{(x_{1},y_{1}),...,(x_{n},y_{n})\\}$ and model to test
$M$, model used to rephrase $M_{p}$.

2: for $i=1,2...,n$ do

3: $x^{{}^{\prime}}_{i}\leftarrow M_{p}(x_{i})$

4: $c_{i}\leftarrow\mathcal{P}(Yes|x_{i},M(x_{i}),M)$

5:
$c_{i}^{{}^{\prime}}\leftarrow\mathcal{P}(Yes|x^{{}^{\prime}}_{i},M(x^{{}^{\prime}}_{i}),M)$

6: end for

7: $\bar{d}\leftarrow\dfrac{\sum_{i=1}^{n}c_{i}-c_{i}^{{}^{\prime}}}{n}$

8: $s_{d}\leftarrow\sqrt{\dfrac{1}{n-1}\sum_{i=1}^{n}(d_{i}-\bar{d})^{2}}$

9: $t\leftarrow\dfrac{\bar{d}}{\frac{s_{d}}{\sqrt{n}}}$, Calculate $p$
according to $t$ and $n$

10: if p < 0.05 (Significant) then

11: Return: $D$ is Contaminated

12: else

13: Return: $D$ is not Contaminated

14: end if

##  5 Experiments

Model |  Method |  Trained Data |  Untrained Data
---|---|---|---
Llama (Contaminated) |  Guided-Prompting |  0.99 |  0.62
PaCoST(simplified) |  0.94 |  0.99
PaCoST(ours) |  6e-8 |  0.92
Mistral (Contaminated) |  Guided-Prompting |  0.99 |  0.99
PaCoST(simplified) |  0.02 |  0.36
PaCoST(ours) |  2e-4 |  0.75
Llama (Original) |  Guided-Prompting |  1e-10 |  1e-9
PaCoST(simplified) |  0.78 |  0.87
PaCoST(ours) |  0.12 |  0.92
Mistral (Original) |  Guided-Prompting |  7e-5 |  1e-3
PaCoST(simplified) |  0.18 |  0.46
PaCoST(ours) |  0.46 |  0.72
Table 2: Main results of intentional contamination. The values are p-value of
the methods, where $p<0.05$ represents statistically significant and probably
contaminated and $p\geq 0.05$ represents un-contaminated. The bold p-values
represents significant results. The underlined values represent false positive
or false negative results.

###  5.1 Intentional Contamination Experiments

First, to validate the effectiveness of our method, we conduct intentional
contamination experiments.

#### Experiment Settings

For these experiments, we select Mistral-7B-Instruct-v0.2(Jiang et al., 2023a)
and Llama-2-7B-Chat(Touvron et al., 2023) as the target models and utilize a
newly released dataset WMDP Li et al. (2024) for intentional contamination.
This dataset, including 3,668 multiple-choice questions about knowledge in
biology, chemistry and cyber, is released in May 2024, ensuring that the
selected models have not been contaminated on this data before.

We conduct supervised fine-tuning (following the second contamination type) on
two models. Though there are two contamination types as we introduced in
Section 3.2, we mainly conduct intentional contamination experiments following
the second contamination type because it is less discussed and somehow more
difficult to detect because it has less trained parts. It is worth mentioning
that our method works properly under the first contamination type, as is shown
in Appendix C.

We sample 1000 samples from biology split from the WMDP dataset to produce
contaminated versions of Llama and Mistral. 400 samples are sampled from the
remaining data in the WMDP dataset to form "clean" (untrained) data. The
choice of number of samples are just for simplicity and does not affect the
final results as we will show later.

For baseline comparisons, given the limited availability of benchmark-level
contamination detection methods, we selected Guided-Prompting (Golchin and
Surdeanu, 2023b) as our baseline. Since Guided-Prompting also utilizes
p-values as an indicator of contamination, this allows for a fair comparison
between our method and theirs.

We also compare the performance of our method with a simplified version that
directly uses ground truth answer to calculate confidence instead of the
model’s generated reponse. Details of the simplified version will be discussed
in Appendix A.

Additionally, we conduct experiments to evaluate the performance of DCQ
(Golchin and Surdeanu, 2023a) and Min-k% Prob (Shi et al., 2024) and find that
they do not perform well for detecting benchmark contamination. A detailed
discussion of these findings can be found in Appendix B.

#### Results and Analysis

The results are presented in Table 2. Our method successfully identifies
contaminated datasets in contaminated models, demonstrated by significant
results on trained data in these models. Importantly, our method avoids false
positives, as it does not return significant results on uncontaminated
datasets, even when applied to contaminated models. For original models, which
are free from contamination, all results are insignificant. These findings
underscore the effectiveness of our method in accurately detecting data
contamination.

In contrast, Guided-Prompting fails to identify contamination in contaminated
models, likely because the instruction part was not included in the training
parts, preventing Guided-Prompting from replicating the original data
accurately. Similarly, the simplified version of our method performs much
better than Guided-Prompting, but it still suffers from false negative
problems. These comparisons further reveal the effectiveness of our method.
Some detailed discussions about this result can be found in Appendix A.

#### Stability under Different Number of Samples

Different datasets vary in the amount of data they contain, and for very large
datasets, it is more practical to sample a subset for contamination detection.
Therefore, it is crucial to validate that our method performs well with
varying sample sizes. To test this, we conducted experiments under the same
settings as above but with different numbers of samples. The results are
presented in Table 3.

Data |  #Sample |  Llama / Mistral (Contaminated) |  Llama / Mistral (Original)
---|---|---|---
Trained Data | 1000 |  6e-8 / 2e-4 | 0.12 / 0.46
500 |  1e-5 / 7e-8 | 0.41 / 0.55
100 |  0.02 / 1e-3 | 0.81 / 0.38
Untrained Data | 400 | 0.92 / 0.75 | 0.92 / 0.72
200 | 0.54 / 0.84 | 0.83 / 0.56
100 | 0.88 / 0.62 | 0.59 / 0.27
Table 3: p-value of different number of samples. The significant results are
in bold.

As indicated by the results, our method works properly with sample sizes
ranging from 100 to 1000, without generating any false positives or false
negatives. This demonstrates the stability of our method across different
sample sizes and highlights that it only requires a subset of the dataset to
effectively detect contamination, thereby reducing the cost of processing
entire datasets.

We do not discuss samples with fewer than 100 instances for two reasons.
First, because our method relies on statistical analysis, a small sample size
can introduce significant randomness, which could interfere with accurate
contamination detection. Second, datasets with fewer than 100 samples are
rare, making the analysis of such scenarios less relevant and meaningful.

We also conducted additional studies to assess the behavior of our method
under various conditions. We demonstrated that our method maintains stable
performance when using different rephrase models $M_{p}$. It is also robust to
reasonable randomness, as it delivers consistent performance under different
random seeds. Furthermore, our method effectively handles various types of
contamination. These findings collectively highlight the superiority of our
method. Detailed discussions can be found in Appendix C.

Model | Arc-c | Arc-e | MMLU | HellaS | WinoG | T-QA
---|---|---|---|---|---|---
Model I | 0.46 | 0.53 | 2e-3 | 3e-8 | 0.78 | 0.57
Model II | 0.18 | 0.30 | 3e-7 | 7e-3 | 0.59 | 0.82
Model III | 1e-3 | 1e-3 | 0.30 | 0.37 | 2e-3 | 0.04
Model IV | 0.02 | 0.28 | 0.09 | 0.59 | 0.25 | 2e-3
Model V | 4e-4 | 7e-4 | 0.71 | 0.63 | 0.10 | 0.20
Model VI | 1e-3 | 0.01 | 0.02 | 0.15 | 3e-8 | 0.24
Model VII | 0.11 | 5e-3 | 0.73 | 0.03 | 0.11 | 0.82
Model VIII | 0.09 | 0.04 | 0.02 | 0.10 | 0.26 | 0.44
Model IX | 0.44 | 0.02 | 0.54 | 3e-13 | 4e-3 | 2e-8
Model X | 0.95 | 0.38 | 0.17 | 0.61 | 0.65 | 0.46
Table 4: p-values of open-source models on widely tested benchmarks. (HellaS:
HellaSwag, WinoG: WinoGrande, T-QA: TruthfulQA)

###  5.2 Tests on Existing LLMs and Benchmarks

After showing the feasibility of our proposed method, we apply it to a variety
of existing popular LLMs and benchmarks to assess their contamination status.
In this section, we introduce the tested benchmarks, models, and present the
experimental results and discussions. Since some benchmarks are extremely
large, we randomly sample 400 samples in each benchmark for detection.

#### Datasets

We conduct benchmark contamination detection experiments on some popular
benchmarks, including MMLU Hendrycks et al. (2020), HellaSwag (Zellers et al.,
2019), Arc-E, Arc-C Clark et al. (2018), TruthfulQA Lin et al. (2021),
WinoGrande Sakaguchi et al. (2021).

#### Models

We select the following open-source LLMs for experiments: Llama-2-Chat (7B,
13B) Touvron et al. (2023), Llama-3-Instruct (8B) AI@Meta (2024), Mistral-
Instruct (7B) Jiang et al. (2023a), Phi-3 (3.8B) Abdin et al. (2024), Qwen1.5
(0.5B, 7B), Qwen2 (7B) Bai et al. (2023), Yi (6B) AI et al. (2024), DeepSeek
(7B) DeepSeek-AI (2024).

#### Evaluation Results

We show the evaluation results in Table 4. To avoid potential harmful effects,
we do not show the names of the models in the results and represent them as
Model I to Model X. Some observations can be drawn from the results.

First of all, all benchmarks are suspected contaminated more or less on
different models. Some benchmarks, like Arc-e, is suspected severely
contaminated. Other benchmarks are are also suspected contaminated and we do
not find a benchmark that is "clean" on all models.

Secondly, almost all models are suspected contaminated more or less on
different benchmarks. Some models, like Model VI, Model IX, are suspected
contaminated on 4 benchmarks out of 6 we tested. Other models are also
suspected contaminated on 2 or 3 benchmarks out of 6 we tested. Model X is
perhaps the "cleanest" model as we do not find significant evidence of
contamination.

###  5.3 Discussion

This result further underscores the urgency of addressing the benchmark
contamination problem in LLM evaluation. As evidenced, almost all models and
benchmarks exhibit varying degrees of suspected contamination. This
contamination undermines the trustworthiness of evaluation results on popular
benchmarks, posing significant challenges for both users and developers.

It is important to note that we do not intend to accuse any LLM provider of
intentional contamination. As previously discussed, given the vast amount of
data required to train LLMs, excluding or even simply detecting benchmark data
within training datasets is an exceedingly difficult task. We must acknowledge
that benchmark contamination may be inevitable due to these constraints.

Instead, we would like to propose two key insights. First, detecting benchmark
contamination is crucial because it allows us to assess whether evaluation
results are trustworthy. While contamination does not inherently imply that a
model is ineffective, recognizing its presence can prompt us to seek
alternative evaluation metrics. This ensures that we are not misled by
artificially high scores, and helps maintain the integrity and reliability of
model evaluations.

Secondly, using specific benchmarks for evaluation may not be suitable. As our
findings reveal, all benchmarks are suspected contaminated to some degree. As
soon as a new benchmark is made public, it quickly becomes susceptible to
contamination because LLMs require large-scale, high-quality data for
training, and benchmarks naturally fit this criterion. However, if a benchmark
is not released publicly, its quality and the evaluation results derived from
it cannot be fully trusted, leading to a dilemma.

Therefore, we advocate for a new LLM evaluation approach that does not rely on
static benchmarks but rather on flexible and dynamic data sources. For
instance, evaluating LLMs based on user feedback data, could provide a dynamic
and resilient measure of model performance. Further, quantitative LLM
evaluation can also be made public - everyone can build his own benchmark for
evaluation. If the results of this large-scale benchmarks could be combined,
the evaluation of LLMs will be more trustworthy and comprehensive.

##  6 Conclusion

In this work, we introduce the issue of benchmark contamination in LLMs and
propose several essential criteria that an effective benchmark contamination
detection method should meet. We highlight that all existing detection methods
fall short of satisfying all of these requirements. We then propose a
benchmark contamination detection method named PaCoST, which uses
significantly higher confidence scores as an indicator of contamination. We
conduct various experiments to demonstrate the effectiveness of our method.
Additionally, we apply our method to popular LLMs and benchmarks and reveal a
significant problem of benchmark contamination across almost all benchmarks
and LLMs we examined.

##  7 Limitations

Our method focuses on detecting benchmark-level contamination and is not
suitable for identifying instance-level contamination. Additionally, our
method involves multiple interactions with the LLM, including one for
paraphrasing, two for answer generation, and two for confidence estimation.
This can result in lower efficiency compared to other approaches.

Moreover, our method requires access to the probability distribution for
confidence estimation, which is not available in black-box LLMs. As a result,
our approach cannot be used to detect benchmark contamination in black-box
LLMs where internal outputs like probability distributions are not accessible.

##  8 Ethics Statement

We honestly report the p-values for various open-source LLMs and benchmarks
without any alteration to enhance or detract from the results. The
intentionally contaminated checkpoints used in our research are for academic
purposes only and will not be released because WMDP is a "dangerous" dataset
that should be forgotten instead of memorized by models. The aim of this work
is to highlight and address the issue of benchmark contamination, not to
promote contamination or criticize any parties involved. We deeply respect the
contributions of LLM and benchmark providers and believe that the problem of
benchmark contamination will be effectively addressed in due course. ChatGPT
is used only to assist writing.

## References

  * Abdin et al. (2024) Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Qin Cai, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Yen-Chun Chen, Yi-Ling Chen, Parul Chopra, Xiyang Dai, Allie Del Giorno, Gustavo de Rosa, Matthew Dixon, Ronen Eldan, Victor Fragoso, Dan Iter, Mei Gao, Min Gao, Jianfeng Gao, Amit Garg, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Jamie Huynh, Mojan Javaheripi, Xin Jin, Piero Kauffmann, Nikos Karampatziakis, Dongwoo Kim, Mahoud Khademi, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Yunsheng Li, Chen Liang, Lars Liden, Ce Liu, Mengchen Liu, Weishung Liu, Eric Lin, Zeqi Lin, Chong Luo, Piyush Madan, Matt Mazzola, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Corby Rosset, Sambudha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Swadheen Shukla, Xia Song, Masahiro Tanaka, Andrea Tupini, Xin Wang, Lijuan Wang, Chunyu Wang, Yu Wang, Rachel Ward, Guanhua Wang, Philipp Witte, Haiping Wu, Michael Wyatt, Bin Xiao, Can Xu, Jiahang Xu, Weijian Xu, Sonali Yadav, Fan Yang, Jianwei Yang, Ziyi Yang, Yifan Yang, Donghan Yu, Lu Yuan, Chengruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou. 2024.  Phi-3 technical report: A highly capable language model locally on your phone.  _Preprint_ , arXiv:2404.14219.
  * AI et al. (2024) 01\. AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. 2024.  Yi: Open foundation models by 01.ai.  _Preprint_ , arXiv:2403.04652.
  * AI@Meta (2024) AI@Meta. 2024.  Llama 3 model card.
  * Bai et al. (2023) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023.  Qwen technical report.  _arXiv preprint arXiv:2309.16609_.
  * Brown et al. (2020a) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020a.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Brown et al. (2020b) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020b.  Language models are few-shot learners.  In _Proceedings of the 34th International Conference on Neural Information Processing Systems_ , NIPS ’20, Red Hook, NY, USA. Curran Associates Inc.
  * Carlini et al. (2021a) Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, and Colin Raffel. 2021a.  Extracting training data from large language models.  _Preprint_ , arXiv:2012.07805.
  * Carlini et al. (2021b) Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, and others. 2021b.  Extracting training data from large language models.  In _30th USENIX Security Symposium (USENIX Security 21)_ , pages 2633–2650.
  * Chen and Mueller (2023) Jiuhai Chen and Jonas Mueller. 2023.  Quantifying uncertainty in answers from any language model via intrinsic and extrinsic confidence assessment.  _arXiv preprint arXiv:2308.16175_.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_.
  * Chowdhery et al. (2023) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023.  Palm: Scaling language modeling with pathways.  _Journal of Machine Learning Research_ , 24(240):1–113.
  * Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018.  Think you have solved question answering? try arc, the ai2 reasoning challenge.  _arXiv preprint arXiv:1803.05457_.
  * Cummings et al. (2023) Rachel Cummings, Damien Desfontaines, David Evans, Roxana Geambasu, Matthew Jagielski, Yangsibo Huang, Peter Kairouz, Gautam Kamath, Sewoong Oh, Olga Ohrimenko, and others. 2023.  Challenges towards the next frontier in privacy.
  * DeepSeek-AI (2024) DeepSeek-AI. 2024.  Deepseek llm: Scaling open-source language models with longtermism.  _arXiv preprint arXiv:2401.02954_.
  * Dong et al. (2024) Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, and Ge Li. 2024.  Generalization or memorization: Data contamination and trustworthy evaluation for large language models.  _arXiv preprint arXiv:2402.15938_.
  * Golchin and Surdeanu (2023a) Shahriar Golchin and Mihai Surdeanu. 2023a.  Data contamination quiz: A tool to detect and estimate contamination in large language models.  _arXiv preprint arXiv:2311.06233_.
  * Golchin and Surdeanu (2023b) Shahriar Golchin and Mihai Surdeanu. 2023b.  Time travel in llms: Tracing data contamination in large language models.  _arXiv preprint arXiv:2308.08493_.
  * Gupta et al. (2022) Samyak Gupta, Yangsibo Huang, Zexuan Zhong, Tianyu Gao, Kai Li, and Danqi Chen. 2022.  Recovering private text in federated learning of language models.  35:8130–8143.
  * Hendrycks et al. (2020) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020.  Measuring massive multitask language understanding.  _arXiv preprint arXiv:2009.03300_.
  * Jagielski et al. (2020) Matthew Jagielski, Jonathan Ullman, and Alina Oprea. 2020.  Auditing differentially private machine learning: How private is private sgd?  33:22205–22216.
  * Jayaraman and Evans (2019) Bargav Jayaraman and David Evans. 2019.  Evaluating differentially private machine learning in practice.  In _28th USENIX Security Symposium (USENIX Security 19)_ , pages 1895–1912.
  * Jiang et al. (2023a) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023a.  Mistral 7b.  _arXiv preprint arXiv:2310.06825_.
  * Jiang et al. (2023b) Mingjian Jiang, Yangjun Ruan, Sicong Huang, Saifei Liao, Silviu Pitis, Roger Baker Grosse, and Jimmy Ba. 2023b.  Calibrating language models via augmented prompt ensembles.
  * Kadavath et al. (2022) Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022.  Language models (mostly) know what they know.  _arXiv preprint arXiv:2207.05221_.
  * Kuhn et al. (2023) Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023.  Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation.  _arXiv preprint arXiv:2302.09664_.
  * Li et al. (2024) Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D. Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, Gabriel Mukobi, Nathan Helm-Burger, Rassin Lababidi, Lennart Justen, Andrew B. Liu, Michael Chen, Isabelle Barrass, Oliver Zhang, Xiaoyuan Zhu, Rishub Tamirisa, Bhrugu Bharathi, Adam Khoja, Zhenqi Zhao, Ariel Herbert-Voss, Cort B. Breuer, Samuel Marks, Oam Patel, Andy Zou, Mantas Mazeika, Zifan Wang, Palash Oswal, Weiran Liu, Adam A. Hunt, Justin Tienken-Harder, Kevin Y. Shih, Kemper Talley, John Guan, Russell Kaplan, Ian Steneker, David Campbell, Brad Jokubaitis, Alex Levinson, Jean Wang, William Qian, Kallol Krishna Karmakar, Steven Basart, Stephen Fitz, Mindy Levine, Ponnurangam Kumaraguru, Uday Tupakula, Vijay Varadharajan, Yan Shoshitaishvili, Jimmy Ba, Kevin M. Esvelt, Alexandr Wang, and Dan Hendrycks. 2024.  The wmdp benchmark: Measuring and reducing malicious use with unlearning.  _Preprint_ , arXiv:2403.03218.
  * Li (2023) Yucheng Li. 2023.  Estimating contamination via perplexity: Quantifying memorisation in language model evaluation.  _arXiv preprint arXiv:2309.10677_.
  * Lin et al. (2021) Stephanie Lin, Jacob Hilton, and Owain Evans. 2021.  Truthfulqa: Measuring how models mimic human falsehoods.  _arXiv preprint arXiv:2109.07958_.
  * Lin et al. (2022) Stephanie Lin, Jacob Hilton, and Owain Evans. 2022.  Teaching models to express their uncertainty in words.  _arXiv preprint arXiv:2205.14334_.
  * Lin et al. (2023) Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2023.  Generating with confidence: Uncertainty quantification for black-box large language models.  _arXiv preprint arXiv:2305.19187_.
  * Mahloujifar et al. (2021) Saeed Mahloujifar, Huseyin A Inan, Melissa Chase, Esha Ghosh, and Marcello Hasegawa. 2021.  Membership inference on word embedding and beyond.
  * Mattern et al. (2023) Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schoelkopf, Mrinmaya Sachan, and Taylor Berg-Kirkpatrick. 2023.  Membership inference attacks against language models via neighbourhood comparison.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 11330–11343, Toronto, Canada. Association for Computational Linguistics.
  * Mireshghallah et al. (2022) Fatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal, Taylor Berg-Kirkpatrick, and Reza Shokri. 2022.  Quantifying privacy risks of masked language models using membership inference attacks.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 8332–8347. Association for Computational Linguistics.
  * Nasr et al. (2023) Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito, Christopher A Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee. 2023.  Scalable extraction of training data from (production) language models.  _arXiv preprint arXiv:2311.17035_.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.  _Preprint_ , arXiv:2303.08774.
  * Oren et al. (2023) Yonatan Oren, Nicole Meister, Niladri Chatterji, Faisal Ladhak, and Tatsunori B Hashimoto. 2023.  Proving test set contamination in black box language models.  _arXiv preprint arXiv:2310.17623_.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Sainz et al. (2023a) Oscar Sainz, Jon Ander Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. 2023a.  Nlp evaluation in trouble: On the need to measure llm data contamination for each benchmark.  _arXiv preprint arXiv:2310.18018_.
  * Sainz et al. (2023b) Oscar Sainz, Jon Ander Campos, Iker García-Ferrero, Julen Etxaniz, and Eneko Agirre. 2023b.  Did chatgpt cheat on your test?  https://hitz-zentroa.github.io/lm-contamination/blog/.  Accessed: 2023-07-06.
  * Sakaguchi et al. (2021) Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021.  Winogrande: An adversarial winograd schema challenge at scale.  _Communications of the ACM_ , 64(9):99–106.
  * Shejwalkar et al. (2021) Virat Shejwalkar, Huseyin A. Inan, Amir Houmansadr, and Robert Sim. 2021.  Membership inference attacks against NLP classification models.  In _NeurIPS 2021 Workshop Privacy in Machine Learning_.
  * Shi et al. (2024) Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. 2024.  Detecting pretraining data from large language models.  In _The Twelfth International Conference on Learning Representations_.
  * Shokri et al. (2017) Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017.  Membership inference attacks against machine learning models.  In _2017 IEEE symposium on security and privacy (SP)_ , pages 3–18. IEEE.
  * Song and Shmatikov (2019) Congzheng Song and Vitaly Shmatikov. 2019.  Auditing data provenance in text-generation models.  In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 196–206.
  * Tian et al. (2023) Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. 2023.  Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback.  _arXiv preprint arXiv:2305.14975_.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _Preprint_ , arXiv:2307.09288.
  * Wang et al. (2019) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019.  Superglue: A stickier benchmark for general-purpose language understanding systems.  _arXiv preprint arXiv:1905.00537_.
  * Wang et al. (2022) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022.  Self-consistency improves chain of thought reasoning in language models.  _arXiv preprint arXiv:2203.11171_.
  * Wei et al. (2022) Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022.  Finetuned language models are zero-shot learners.  _Preprint_ , arXiv:2109.01652.
  * Xiong et al. (2023) Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. 2023.  Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms.  _arXiv preprint arXiv:2306.13063_.
  * Yang et al. (2023) Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, Joseph E Gonzalez, and Ion Stoica. 2023.  Rethinking benchmark and contamination for language models with rephrased samples.  _arXiv preprint arXiv:2311.04850_.
  * Yeom et al. (2018) Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. 2018.  Privacy risk in machine learning: Analyzing the connection to overfitting.  In _2018 IEEE 31st Computer Security Foundations Symposium (CSF)_ , pages 268–282.
  * Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019.  Hellaswag: Can a machine really finish your sentence?  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_.
  * Zhang et al. (2024) Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Dylan Slack, Qin Lyu, Sean Hendryx, Russell Kaplan, Michele Lunati, and Summer Yue. 2024.  A careful examination of large language model performance on grade school arithmetic.  _Preprint_ , arXiv:2405.00332.
  * Zhang* et al. (2020) Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020.  Bertscore: Evaluating text generation with bert.  In _International Conference on Learning Representations_.
  * Zhao et al. (2023) Yukun Zhao, Lingyong Yan, Weiwei Sun, Guoliang Xing, Chong Meng, Shuaiqiang Wang, Zhicong Cheng, Zhaochun Ren, and Dawei Yin. 2023.  Knowing what llms do not know: A simple yet effective self-detection method.  _arXiv preprint arXiv:2310.17918_.

##  Appendix A Discussions about Intentional Contamination Experiment

#### Details about Simplified Version of Our Method

We briefly introduce the simplified version of our method. Recall that our
method calculate confidence $c_{i}=\mathcal{P}(Yes|x_{i},M(x_{i}),M)$ for a
given instance $(x_{i},y_{i})$. But it is natural to question whether it is
possible to use $\widetilde{c}_{i}=\mathcal{P}(Yes|x_{i},y_{i},M)$, that is,
to directly calculate model’s "confidence" towards the ground truth answer. So
we design a simplified version of our method in Algorithm 2.

Algorithm 2 PaCoST(Simplified)

1: Input benchmark $D=\\{(x_{1},y_{1}),...,(x_{n},y_{n})\\}$ and model to test
$M$, model used to rephrase $M_{p}$.

2: for $i=1,2...,n$ do

3: $x^{{}^{\prime}}\leftarrow M_{p}(x_{i})$

4: $\widetilde{c}_{i}\leftarrow\mathcal{P}(Yes|x_{i},y_{i},M)$

5:
$\widetilde{c}_{i}^{{}^{\prime}}\leftarrow\mathcal{P}(Yes|x^{{}^{\prime}}_{i},y_{i},M)$

6: end for

7:
$\bar{d}\leftarrow\dfrac{\sum_{i=1}^{n}\widetilde{c}_{i}-\widetilde{c}_{i}^{{}^{\prime}}}{n}$

8: $s_{d}\leftarrow\sqrt{\dfrac{1}{n-1}\sum_{i=1}^{n}(d_{i}-\bar{d})^{2}}$

9: $t\leftarrow\dfrac{\bar{d}}{\frac{s_{d}}{\sqrt{n}}}$, Calculate $p$
according to $t$ and $n$

10: if p < 0.05 (Significant) then

11: Return: $D$ is Contaminated

12: else

13: Return: $D$ is not Contaminated

14: end if

#### Discussion about Guided-Prompting

Surprisingly, we find that Guided-Prompting generates numerous false-positive
results on uncontaminated models. Since the WMDP dataset was released after
the model checkpoints were created, and given that WMDP was authored by human
experts (Li et al., 2024), it is highly unlikely that WMDP was initially
contaminated. Even if it were initially contaminated, Guided-Prompting should
have been able to detect this in the subsequently contaminated checkpoints,
which it failed to do. This observation further supports our assertion that
Guided-Prompting is unstable across different prompts. The significance
indicated by Guided-Prompting may stem from this instability rather than from
genuine contamination.

#### Discussion about Simplified Version

The simplified version of our method works much better than Guided-Prompting,
as it correctly identifies one contaminated case and all un-contaminated
cases. However, it makes a false negative mistake on contaminated Llama,
making it less effective compared with the original version of PaCoST.

We would like to attribute this false negative to the same reason mentioned in
Yang et al. (2023), which argues that contaminated models would bear similar
high performance even on rephrased samples. Therefore, using ground-truth
answer may result in contaminated model behaving similarly on original samples
and rephrased samples, leading to false negative mistakes. In contrast, our
focus is that model will be more confident when answering the question instead
of towards the correct answer. As can be seen from results in Table 2, this
assumption is more accurate and works better. Though the simplified version
works well under some circumstances, our whole PaCoST performs better.

##  Appendix B Comparison with Other Methods

There are also many methods aiming at detecting contamination that are worth
discussing. We mainly discuss two of them: DCQ (Golchin and Surdeanu, 2023a)
and Min-k% Prob (Shi et al., 2024).

#### Discussion of DCQ

DCQ is a replication-based method which posits that models can distinguish
between data they have been trained on and similar data they have not
encountered during training. This method employs a multiple-choice quiz to
detect contamination. We apply this method in our experiments and reported the
accuracy in Table 5.

Model |  Trained Data |  Untrained Data
---|---|---
Llama (Cont.) | 0.5 | 0.39
Table 5: Accuracy of DCQ. Cont. represents contaminated.

As evident from the results, the accuracy is even worse than random
guessing—random guessing would yield an accuracy of approximately 0.5. We
believe this outcome is due to the following reasons.

First of all, the contaminated Llama follows the second contamination type,
where only the answer part, not the instruction part, is trained. However, DCQ
requires the model to identify the exact instruction part from multiple
choices, which is particularly challenging given that the instruction part was
not part of the training. This mismatch likely contributes to the method’s
poor performance in our experiments.

Secondly, numerous studies have demonstrated that LLMs are highly sensitive to
prompts, and the order of choices in a multiple-choice question can
significantly influence the outcome. This sensitivity leads to considerable
variability in the method’s performance, making it unreliable. As a result,
users cannot draw definitive conclusions from its results due to this inherent
instability.

#### Discussion of Min-k% Prob

Min-k% Prob focuses on the k% tokens with the smallest probabilities and $k$
is set to 20 to achieve the best performance according to Shi et al. (2024).
This method has two problems. The first one is, traditional Min-k% Prob also
requires the instruction to be trained. However, we do can adapt this method
to only work on the answer part of a piece of data. But for relatively short
trained parts (like an answer), 20% tokens are simply one or two tokens, which
may introduce too much randomness. The second one is, it requires a pre-
defined threshold to determine contamination, but this threshold is hard to
choose.

We report the accuracy of Min-k% Prob in Table 6. We select $k=20$ and
threshold $\epsilon=0.1$. Specifically, if and only if the average probability
of the min-20% tokens is larger than 0.1, we classify the instance as
contaminated. We present the accuracy results for both the original Min-k%
Prob and our adapted version of Min-k% Prob.

Method | Model | Trained | Untrained
---|---|---|---
Min-k% Prob (Original) | Llama(Cont.) | 0.02 | 0.97
Llama(Original) | 0.94 | 0.98
Min-k% Prob (Adapted) | Llama(Cont.) | 0.86 | 0.6
Llama(Original) | 1.0 | 1.0
Table 6: Accuracy of Min-k% Prob. Cont. represents contaminated.

There are several interesting observations based on the results. First, the
original Min-k% Prob fails to determine contamination in the contaminated
model because the instruction part is not trained. This aligns with our
previous discussion.

The adapted Min-k% Prob performs much better on both datasets. However, we
observe an interesting phenomenon: for uncontaminated Llama, the model tends
to output a relatively long response, causing the answer itself to have a
relatively small probability, which leads to high detection accuracy. For
contaminated Llama, the model outputs a single choice as response, but the
probability of this choice is very high (e.g., 0.99999) no matter it is
correct or not.

As a result, the contamination detection accuracy essentially becomes the
accuracy of question answering. For contaminated data, if the model correctly
answers a question, it outputs a very high probability, leading Min-k% Prob to
classify it as contaminated. Similarly, for uncontaminated data, if the model
correctly answers a question, it also outputs a very high probability, still
causing Min-k% Prob to classify it as contaminated. Thus, in this case, Min-k%
Prob is effectively detecting whether the question is correctly answered,
rather than whether the question is contaminated.

This observation also highlights the problems of using answer probabilities as
a confidence score or using perplexity to determine contamination. Simple
probabilities are easily influenced by various factors, including formatting,
leading to unreliable results.

##  Appendix C Discussions of Our Method

In this part, we would like to make some detailed discussions about our method
to show that our method provides stable and trustworthy results. For
simplicity, the following experiments are conducted on Llama only.

#### Quality of Rephrasing

Model |  Trained Data |  |  Untrained Data |
---|---|---|---|---
0 |  42 |  302 |  3407 |  9056 |  | 0 | 42 | 302 | 3407 | 9056 |
Llama (Contaminated) | 6e-4 | 4e-5 | 9e-6 | 4e-8 | 0.01 |  | 0.94 | 0.96 | 0.97 | 0.63 | 0.97 |
Llama (Original) | 0.73 | 0.98 | 0.98 | 0.83 | 0.77 |  | 0.99 | 0.92 | 0.99 | 0.99 | 0.99 |
Table 7: p-value of different random seeds. The significant results are in
bold.

Though LLMs are known to handle various tasks effectively, it is still
reasonable to question their proficiency at rephrasing. If the rephrasing
model $M_{p}$ fails to correctly rephrase a question, the results of our
method would become meaningless. Therefore, we aim to investigate the quality
of rephrasing.

Since we primarily use Llama-2-Chat-7B for rephrasing, we focus on evaluating
its rephrasing quality. We use the same dataset split mentioned in Section 5
and randomly sample 100 instances from each split to evaluate the quality of
rephrasing. We use two evaluation methods: BERT-Score (Zhang* et al., 2020)
and human study. We employ two human annotators to check whether each
rephrasing result is correct (i.e., it does not change the original meaning
and is not exactly the same as the original instance) and annotate each as 0
(incorrect) or 1 (correct). The results are shown in Table 8.

Data | BERT-Score | Human Evaluation
---|---|---
Trained | 0.95 | 0.89
Untrained | 0.94 | 0.91
Table 8: Rephrasing quality evaluation average results.

As can be seen from the results, the rephrasing outputs have relatively high
BERT-Score and human evaluation scores. This observation clearly demonstrates
that using Llama-2-Chat-7B for rephrasing is suitable and does not interfere
with contamination detection.

#### Performance Stability: Rephrasing

We choose Llama-2-Chat-7B for rephrasing because it is a powerful model.
However, the rephrasing model $M_{p}$ does not affect the final result as long
as the model is capable enough. To validate our method provides stable results
using different rephrasing models, we use another model,
Mistral-v0.2-Instruct-7B (Jiang et al., 2023a), for rephrasing. Other settings
remain the same as in the previous experiments. The results are shown in Table
9.

Data |  Rephrase Model |  Llama (Contaminated) |  Llama (Original)
---|---|---|---
Trained Data | Llama | 6e-8 | 0.12
Mistral | 2e-3 | 0.99
Untrained Data | Llama | 0.92 | 0.92
Mistral | 0.23 | 0.99
Table 9: p-value of different rephrase models. The significant results are in
bold.

Using either Llama or Mistral for rephrasing does not affect the outcomes,
confirming that we can select any sufficiently powerful model for rephrasing.
We use Llama-2-Chat-7B for rephrasing in our other experiments as mentioned
earlier.

#### Performance Stability: Contamination Types

As is discussed, there are two types of benchmark contamination. Our previous
experiments primarily focus on the second type, as it involves shorter trained
parts and is somewhat harder to detect. However, our method is also capable of
detecting the first type. The results are shown in Table 10.

Data |  Llama (Cont. I) |  Llama (Cont. II)
---|---|---
Trained | 4e-15 | 6e-8
Untrained | 0.75 | 0.92
Table 10: p-value of different contamination types. The significant results
are in bold. Cont. represents contaminated.

As can be seen from the results, our method still works properly under the
first contamination type. This result shows that our method is able to detect
contamination with different types, which further proves its effectiveness.

#### Performance Stability: Randomness

Paraphrasing unavoidably introduces randomness into contamination detection,
so it is necessary to investigate the stability of our method under such
conditions. We conduct this experiment using the same settings as above but
randomly select five seeds to explicitly introduce randomness. The results are
listed in Table 7.

With the random seed changing, the corresponding p-value changes, but the
significance (or in-significance) remains unchanged. This observation clearly
demonstrates the stability of our method under randomness.

##  Appendix D Prompt

Rephrase Prompt Instruction: You are provided with a question. Your task is to
rephrase this question into another question with the same meaning. When
rephrasing the question, you must ensure that you follow the following rules:
(1). You must ensure that you generate a rephrased question as your response.
(2). You must ensure that the rephrased question bears the same meaning with
the original question. Do not miss any information. (3). You must only
generate a rephrased question. Any other information should not appear in your
response. (4). Do not output any explanation. (5). Do not modify the numbers
or quantities in the question. You should remain them unchanged Example:
{In-Context Examples} Input: {input} Output:
Confidence Estimation Prompt Instruction: You are an expert in judging whether
the answer is correct. You will be given a question and a corresponding
answer. Your job is to determine whether this answer is correct. You should
only respond with Yes or No. Example:
{In-Context Examples} Input: {input} Output:
Confidence Estimation Example Instruction: You are an expert in judging
whether the answer is correct. You will be given a question and a
corresponding answer. Your job is to determine whether this answer is correct.
You should only respond with Yes or No. Example:
{In-Context Examples} Input: The question is: At what concentration does
prolonged exposure to phosgene become dangerous? A. 100 ppm B. 25 ppm C. 1 ppm
D. 10 ppm The answer is B. Is the answer correct according to the given
question? Output:
Yes. Output Distribution: $P(Yes)=0.92$, which means confidence $c=0.92$.


# Exploring Lead Free Mixed Halide Double Perovskites Solar Cell

Md Yekra Rahman*, Dr. Sharif Mohammad Mominuzzaman
Department of Electrical & Electronic Engineering
Bangladesh University of Engineering and Technology, Dhaka, Bangladesh
*Corresponding Author Email<EMAIL_ADDRESS>
###### Abstract

The significant surge in energy use and escalating environmental concerns have
sparked worldwide interest towards the study and implementation of solar cell
technology. Perovskite solar cells (PSCs) have garnered remarkable attention
as an emerging third-generation solar cell technology. This paper presents an
in-depth analysis of lead-free mixed halide double perovskites in the context
of their potential uses in solar cell technology. Through the previous studies
of various mixed halide double perovskite materials as potential absorber
layer materials, it has been observed that Cs2TiI6-xBrx possesses promising
characteristics. In this study, simulations were conducted using SCAPS-1D
software to explore all possible combinations for x= 0 to 6. The materials for
the Hole Transport Layer (HTL) and Electron Transport Layer (ETL) and absorber
layer, along with their optimal thicknesses, are selected to yield the most
promising results in terms of open-circuit voltage (Voc), short-circuit
current density (Jsc), fill factor (FF), and power conversion efficiency. A
novel tolerance factor is employed to assess structural stability of
perovskites. FTO/Cu2O/Cs2TiI5Br1/WS2 emerges as the best combination in terms
of the efficiency with 19.03% but FTO/Cu2O/Cs2TiI2Br4/WS2 shows good stability
with 13.31% efficiency.

Keywords—Perovskite Solar Cell, SCAPS-1D, Hole Transport Layer, Electron
Transport Layer, Stability

##  I Introduction

Over the decade, solar cells made from materials like CdTe, CIGS, GaInP, and
CZTS were outperformed by a new kind of solar cell called Perovskite Solar
Cells (PSCs) [1]. These PSCs are made from inorganic-organic halide ABX3 type
structure. It also possess several impressive and distinct photoelectric
properties, such as a high absorption coefficient, long carrier diffusion
length, high carrier mobility, low recombination rate, and tunable band gaps
through chemical modification. [2]. In the past few years, the efficiency in
hybrid halide organic-inorganic perovskite solar cells, specifically
CH3NH3PbI3 (MAPbI3), has improved quickly, rising from 3.8% in 2009 [3] to
26.1% in 2023[4].

Despite the rapid increase in the efficiency, there are still a few major
challenges hindering the widespread adoption of this technology. One issue is
the intrinsic and extrinsic instability of the organic components, and another
significant concern is the toxicity of lead. To address these issues, all-
inorganic cesium halide perovskites are being explored as alternatives[5]. In
these, cesium Cs+ is used to replace MA+ or FA+ in the A-site of the
perovskite structure. These are more stable thermodynamically and less prone
to moisture absorption. Research by Yang et al. indicates that the high-
temperature phase of CsPbBr3 shows greater thermodynamic stability, as well as
intrinsic and extrinsic stability[6]. To make perovskites more stable and
lead-free on the B site, A2BIBIIIX6 structures have been introduced. Building
on this strategy, researchers have successfully synthesized several halide
double perovskites, such as Cs2AgInCl6 [7] and Cs2AgSbCl6 [8]. But these new
perovskites have excessively large band gaps. This makes them unsuitable for
use in single-junction solar cells. Meanwhile, Woodward et. al have developed
the first halide double perovskite Cs2AgBiBr6 which has large band gap and
resistance to air and water exposure [9]. After that, the mutation of B2+ ions
in double perovskites has been achieved by incorporationg tetravalent cations
resulting in ”vacancy-ordered” double perovskites with the general formula
A2BX6. Chen et al. successfully synthesized Cs2TiBr6, with experimental
results showing that it possesses a stable bandgap of 1.8eV[5]. The thin-film
perovskite solar cells based on Cs2TiBr6 have demonstrated a consistent
efficiency of up to 3.3%[10]. Qui et al. conducted a detailed analysis of the
band gaps, spectral characteristics, and formation energies of Cs2Ti(Br1-xYx)6
(where Y represents Cl and I; and x varies from 0 to 1 in increments of 0.25)
by utilizing first-principles methods based on Density Functional Theory (DFT)
calculations [11].

Despite these developments, the challenge of identifying the most effective
ionic substitutions for enhanced efficiency and stability remains. To forecast
the stability of the perovskite structure based on the chemical formula ABX3
Goldschmidt tolerance factor (t) has been used.

$t=\frac{r_{A}+r_{X}}{\sqrt{2}(r_{B}+r_{X})}$ (1)

where ionic radii, $r_{i}$, of each ion (i = A, B, X). For stable perovskite,
$0.825<t<1.059$ But It has only 74% accuracy for 576 experimentally collected
ABX3 type materials[12]. To improve accuracy, a new tolerance factor is
introduced by Bartel et al.

$\tau=\frac{r_{X}}{r_{B}}-n_{A}(n_{A}-\frac{\frac{r_{A}}{r_{B}}}{ln(\frac{r_{A}}{r_{B}})})$
(2)

where $n_{A}$ is the oxidation state. For stable perovskite, $\tau<4.18$ and
the accuracy is 92% for the same database [12].

In this study, we will further explore their effect in performance parameters
of solar cell with Cs2TiI6-xBrx used as absorber layer. It also focuses on
selecting specific materials for each of the other two key layers of PSCs,
with the primary model being the FTO/ETL/Cs2TiI2Br4/HTL. Here Cs2TiI2Br4 is
chosen as initial absorber layer. Various permutations for this study are
depicted in Fig-1, and modifications to the absorber perovskite layer are
explored to assess their impact on performance metrics by using SCAPS-1D.
Finally, the improved tolerance factor ($\tau$) has been used to the
investigation of the absorber layer’s stability to assess the mixed halide
double perovskite structure’s performance with various compositions.

Figure 1: Possible setups of cells

##  II Methodology

SCAPS-1D software requires all relevant factors for equation solving to be
precisely configured. Key material parameters for the simulation include the
bandgap ($E_{g}$) of each layer, electron affinity ($\chi$), relative
dielectric permittivity ($\epsilon_{r}$), effective density of states in the
conduction and valence bands ($N_{c}$ and $N_{v}$), mobilities of electrons
and holes ($\mu_{n}$ and $\mu_{h}$), thermal velocities of electrons and holes
($V_{th,n}$ and $V_{th,h}$). The calculation of the absorption coefficients
for each layer was performed using the built-in model. For the front and back
contacts of the solar cell, gold (Au) and aluminum (Al) were used, with their
work functions set at 5.45 eV and 4.1 eV respectively. The defect density in
each layer was assumed to be around $10^{17}\text{ cm}^{-3}$. The cross-
section for electron capture is measured at $10^{-19}\text{ cm}^{-2}$ for
Perovskite/ETL and $10^{-18}\text{ cm}^{-2}$ for HTL/Perovskite. In contrast,
the cross-section for hole capture stands at $10^{-18}\text{ cm}^{-2}$ for
Perovskite/ETL and $10^{-19}\text{ cm}^{-2}$ for HTL/Perovskite respectively.
Table I summarizes these parameters. The simulations were conducted under
standard operating conditions using solar spectrum AM 1.5G and a temperature
of 300K.

TABLE I: Initial material parameters used for SCAPS simulation [13, 14, 15, 16, 17] Parameter | FTO |  TiO2 |  Cs2TiI2Br4 |  CuSbS2 | CuSCN |  Cu2O |  MoS2 | CuI |  WS2 | ZnO
---|---|---|---|---|---|---|---|---|---|---
Thickness ($\mu$m) | 0.5 | 0.05 | 0.35 | 0.35 | 0.35 | 0.35 | 0.35 | 0.35 | 0.05 | 0.05
Bandgap (eV) | 3.5 | 3.2 | 1.38 | 1.58 | 3.68 | 2.17 | 1.29 | 2.98 | 1.87 | 3.3
Electron affinity (eV) | 4.0 | 3.9 | 4.15 | 4.2 | 1.9 | 3 | 4.2 | 2.1 | 4.3 | 4.1
Dielectric permittivity | 9 | 38 | 6.6 | 8.2 | 10 | 7.5 | 3 | 6.5 | 11.9 | 9
Density of states at CB (cm-3) |  2.2$\times 10^{18}$ |  2.2$\times 10^{18}$ |  6$\times 10^{19}$ |  2$\times 10^{18}$ |  2.2$\times 10^{18}$ |  2$\times 10^{18}$ |  2.2$\times 10^{18}$ |  2.8$\times 10^{19}$ |  1$\times 10^{19}$ |  2.2$\times 10^{18}$
Density of states at VB (cm-3) |  1.8$\times 10^{19}$ |  1.8$\times 10^{19}$ |  2.4$\times 10^{19}$ | $10^{19}$ |  1.8$\times 10^{19}$ |  1.1$\times 10^{19}$ |  1.8$\times 10^{19}$ |  1.1$\times 10^{19}$ |  2.4$\times 10^{19}$ |  1.9$\times 10^{19}$
Electron and Hole thermal velocity |  107 |  107 |  107 |  107 |  107 |  107 |  107 |  107 |  107 |  107
Electron mobility (cm2V-1s-1) | 20 | 20 | 8.16 | 4.9 |  10-4 | 200 | 100 |  1.7$\times 10^{-4}$ | 260 | 100
Hole mobility (cm2V-1s-1) | 10 | 10 | 8 | 4.9 |  10-2 | 80 | 150 |  2$\times 10^{-4}$ | 51 | 25
Donor Doping Density (cm-3) |  1019 |  1016 |  1018 | 0 | 0 | 0 | 0 | 0 |  1017 |  1017
Acceptor Doping Density (cm-3) | 0 | 0 |  1018 |  1018 |  1017 |  2$\times 10^{19}$ |  1018 |  1018 | 0 | 0
Defect Density (cm-3) |  1015 |  1017 |  1015 |  1017 |  1017 |  1017 |  1017 |  1017 |  1017 |  1017

##  III Results and Discussion

###  III-A J-V Characteristics for different HTLs and varying their thickness

The J-V characteristics for seven HTLs - $\text{CuSbS}_{2}$, CuSCN,
$\text{Cu}_{2}\text{O}$, CuI, Spiro-OMETAD, $\text{MoO}_{2}$, and
$\text{MoS}_{2}$ were investigated to assess their impact on cell performance.
$\text{CuSbS}_{2}$ shows the poorest performance, with minimum Jsc and Voc.
So, it’s not optimal for high-efficiency PSCs. CuSCN, $\text{Cu}_{2}\text{O}$,
CuI, Spiro-OMETAD, and $\text{MoO}_{2}$ have similar Jsc as displayed in
Fig-2. However, $\text{MoS}_{2}$ outperforms all in terms of Jsc. Spiro-OMETAD
and $\text{MoS}_{2}$ have also similar Voc. But $\text{Cu}_{2}\text{O}$
displays higher Jsc and Voc for efficient charge transport and separation in
PSCs.

Figure 2: J-V characteristics for different HTLs

Moreover, in terms of all thicknesses from 0.1$\mu m$ to 1$\mu m$ it is found
that $\text{Cu}_{2}\text{O}$ and $\text{MoO}_{2}$ demonstrate higher
efficiencies compared to other HTL materials (Fig-3a). $\text{Cu}_{2}\text{O}$
maintains the highest efficiency and CuI shows a decline in efficiency with
increasing thickness.

(a)

(b)

(c)

(d)

Figure 3: Performance parameters of various HTLs at different thicknesses. (a)
Efficiency (b) Open-circuit voltage (c) Short-circuit current density (d) Fill
factor

$\text{CuSbS}_{2}$ displays the lowest efficiency at all
thicknesses.Furthermore, examining Fig-3(d), it can be observed that
$\text{Cu}_{2}\text{O}$ exhibits the highest FF. Therefore, based on this
assessment of efficiency and FF, $\text{Cu}_{2}\text{O}$ was identified as the
most appropriate HTL material for this study.

###  III-B J-V Characteristics for different ETLs and varying their thickness

The J-V characteristics of five ETLs, specifically $\text{TiO}_{2}$, ZnO, CdS,
$\text{WS}_{2}$, and $\text{SnO}_{2}$, were evaluated to assess their efficacy
in PSCs. By performing comparative analysis of the curves, as depicted in
Fig-4, it can be seen that there is a lack of substantial variation among the
ETLs. Consequently, determining an optimal ETL from the materials poses a
challenge.

Figure 4: J-V characteristics for different ETLs

The Y-axis of Fig-5 across all the curves indicates a negligible impact of
layer thickness of the ETLs on the performance parameters. A closer look in
the zoomed-in plot of Fig-5(a), reveals that $\text{WS}_{2}$ shows higher
efficiency at a thickness of 0.04 µm. Additionally, $\text{WS}_{2}$ also
surpasses the other ETLs in terms of FF. These observations led to the
selection of $\text{WS}_{2}$ as the preferred ETL material with the thickness
of 0.04 µm.

(a)

(b)

(c)

(d)

Figure 5: Performance parameters of various ETLs at different thicknesses. (a)
Efficiency (b) Open-circuit voltage (c) Short-circuit current density (d) Fill
factor

###  III-C Effect of donor and acceptor concentration of ETL, HTL and absorber
layer

Cu2O and WS2 were identified as the respective ETL, HTL from the previous
observations. Cs2TiI2Br4 was selected as initial absorber layer . Four 2D
color maps in each of Fig-6 and Fig-7 delineate the variations in efficiency,
Voc, Jsc, and FF, with color gradations ranging from deep blue (low values) to
yellow (high values).

(a)

(b)

(c)

(d)

Figure 6: Performance parameters for varying acceptors and donors
concentration of Cu2O and WS2. (a) Efficiency (b) Open-circuit voltage (c)
Short-circuit current density (d) Fill factor

Fig-6(a) illustrates that higher efficiency can be attained by increasing both
the acceptor and donor concentration in Cu2O and WS2 respectively. Fig-6(c)
indicates that lower Cu2O acceptor concentrations and higher WS2 donor
concentrations show higher Jsc. Moreover, as shown in Fig-6(d), FF is more
influenced by Cu2O’s acceptor concentration. So, this study adopts an acceptor
concentration of 8.5$\times 10^{19}$ cm-3 for Cu2O and a donor concentration
of 5$\times 10^{18}$ cm-3 for WS2.

(a)

(b)

(c)

(d)

Figure 7: Performance parameters for varying acceptors and donors
concentration of absorber layer. (a) Efficiency (b) Open-circuit voltage (c)
Short-circuit current density (d) Fill factor

As depicted in Fig-7(a), It can be noted that the higher efficiency for
Cs2TiI2Br4 is achieved when the acceptor concentration is around 1019 and the
donor concentration is within the vicinity of 1017. Fig-7(d) indicates that a
high FF can be achieved with an acceptor concentration of about 1019 cm-3 and
a donor concentration ranging between 1016 cm-3 and 1017 cm-3. From the
insights gained from these analyses, the optimal concentration for the
absorber layer was set with an acceptor concentration of 8.9$\times 10^{19}$
cm-3 and a donor concentration of 4.5$\times 10^{17}$ cm-3.

###  III-D Effect of thickness and defect density of absorber layer

Fig-8 and Fig-9 show how the thickness and defect density of the absorber
layer affects solar cell performance.

(a)

(b)

(c)

(d)

Figure 8: Performance parameters for varying thickness of absorber layer. (a)
Efficiency (b) Open-circuit voltage (c) Short-circuit current density (d) Fill
factor

(a)

(b)

(c)

(d)

Figure 9: Performance parameters for varying defect density of absorber layer.
(a) Efficiency (b) Open-circuit voltage (c) Short-circuit current density (d)
Fill factor

In Fig-8(a), an increase in efficiency acn be seen with the increase of
absorber layer thickness up to a certain point. After that the efficiency
begins to diminish. Likewise, an increase in FF up to a thickness of about 3.2
µm, followed by a subsequent decrease can be seen from Fig-8(d). On the other
hand, the efficiency remains relatively unaffected until a defect density of
1012 cm-3 is reached. Beyond that a significant drop in efficiency occurs
(Fig-9(a)). It notably falls from around 29% to 19% as the defect density
escalates from 1012 cm-3 to 1015 cm-3 . This highlights cell’s susceptibility
to higher defect densities in absorber layer. Forthis study, a defect density
of 1015 cm-3 was selected based on practical considerations and empirical
validation.

In contrast, the FF’s response to increasing defect densities, as illustrated
in Fig-9(d), is less severe. The FF decreases marginally from 89.2% to 88.5%
when defect density rises from 1012 to 1015 , indicating a relatively
resilient behavior of FF against variations in defect density within the
specified range. Although the decrease in FF should not be overlooked, its
impact is less significant compared to the effect on efficiency.

###  III-E J-V characteristics and spectral responsivity for Cs2TiI6-xBrx (x=0
to 6)

The influence of varying the mixed halide composition (denoted as x) in
Cs2TiI6-xBrx, ranging from x=0 to 6, on the performance of the solar cell has
been showcased in Fig-10. The analysis of Voc reveals that altering x between
1 and 5 results in minimal Voc variation. However, at the extremes of x=0 and
6, there is a noticeable change in Voc, indicating that these materials
significantly influence Voc.

Figure 10: J-V characteristics for Cs2TiI6-xBrx (x= 0 to 6)

In terms of Jsc, a trend of decreasing Jsc with an increase in x from 1 to 5
was observed to generate electron hole pairs. The combination of these
findings leads to the deduction that Cs2TiI5Br1 possesses the most efficient
absorber layer characteristics, evidenced by the stable Voc and the optimal
Jsc

Figure 11: Spectral responsivity for Cs2TiI6-xBrx (x= 0 to 6)

This study also explored the spectral responsivity. Fig-11 presents the
spectral responsivity of Cs2TiI6-xBrx (x= 0 to 6). A notable observation is
that as there is a rise in bromine concentration, there is a shift in the peak
response wavelength towards the shorter side of the spectrum. Furthermore,
with the decrease in x, the amplitude of this peak also increases.

###  III-F Performance and stability analysis

This paper’s focus on mixed halide double perovskite materials involved a
comparative analysis of their efficiency and stability to identify the most
suitable absorber material for PSCs. The key findings are summarized in Table
II.

It can be seen that an increase in the value of x in (Cs2TiI6-xBrx) leads to a
reduction in efficiency. The material Cs2TiI5Br1 showcased the highest
efficiency among the variants tested. However, its stability, when assessed
using the improved tolerance factor ($\tau$) and the associated stability
probability P($\tau$), is significantly lower compared to other materials.

On the other hand, materials like Cs2TiBr6, Cs2TiI1Br5, and Cs2TiI2Br4, which
have lower efficiencies, exhibit higher stability. Notably, Cs2TiI2Br4
achieves a balance between efficiency (13.31%) and stability, displaying the
highest efficiency among the more stable materials.

Therefore, Cs2TiI2Br4 is identified as a superior absorber material
considering both efficiency and stability and emerges as a promising candidate
for future research and potential application in perovskite solar cell
technology.

TABLE II: Performance Parameters with $\tau$ and P($\tau$) for Various Mixed Halide Double Perovskites | Cs2TiBr6 | Cs2TiI1Br5 | Cs2TiI2Br4 | Cs2TiI3Br3 | Cs2TiI4Br2 | Cs2TiI5Br1 | Cs2TiI6
---|---|---|---|---|---|---|---
Voc (V) | 0.995876 | 0.542262 | 0.556572 | 0.56144 | 0.56598 | 0.568603 | 1.306315
Jsc (mA/cm2) | 12.90499 | 22.88736 | 29.58543 | 33.50023 | 38.48739 | 41.24266 | 15.38558
FF (%) | 77.4376 | 80.1015 | 80.864 | 81.0635 | 81.167 | 81.17 | 87.7209
Efficiency (%) | 9.9521 | 9.9414 | 13.3154 | 15.2467 | 17.6807 | 19.0349 | 17.6305
$\tau$ | 4.074187 | 4.120698 | 4.167210 | 4.213722 | 4.260233 | 4.306745 | 4.353257
P($\tau$) | 0.584539 | 0.562502 | 0.540138 | 0.517977 | 0.495071 | 0.473635 | 0.450955

##  IV Conclusion

In this study, we analyzed the performance and stability of materials used in
ETLs, HTLs, and absorber layer of PSCs. Our investigation revealed CuSbS2 as
the least effective ETL, while Cu2O and MoO2 outperformed other HTLs, with
Cu2O displaying the highest efficiency across various thicknesses. Optimal
charge carrier concentrations were determined, with 8.5$\times 10^{19}$ cm-3
for the acceptor in Cu2O and 5$\times 10^{18}$ cm-3 for the donor in WS2. A
critical outcome was the impact of absorber layer thickness on device
performance. Defect density also emerged as a significant factor, affecting
efficiency more severely than FF. Chosen defect density of 1015 cm-3 in the
absorber layer balanced efficiency and stability considerations. Furthermore,
we explored mixed halide double perovskites, pinpointing Cs2TiI5Br1 as a
potential efficient absorber with a wide absorption range up to 1155 nm. In
the stability analysis Cs2TiI2Br4 emerged as a promising candidate, offering a
balance of high efficiency and good stability. This study provides insights
for developing high-performance and stable PSCs which will contribute
significantly to the advancement of effective solar energy solutions.

## References

  * [1] M. Green, E. Dunlop, J. Hohl-Ebinger, M. Yoshita, N. Kopidakis, and X. Hao, “Solar cell efficiency tables (version 57),” Progress in photovoltaics: research and applications, vol. 29, no. 1, pp. 3–15, 2021.
  * [2] D. Shi, V. Adinolfi, R. Comin, M. Yuan, E. Alarousu, A. Buin, Y. Chen, S. Hoogland, A. Rothenberger, K. Katsiev, Y. Losovyj, X. Zhang, P. A. Dowben, O. F. Mohammed, E. H. Sargent, and O. M. Bakr, “Low trap-state density and long carrier diffusion in organolead trihalide perovskite single crystals,” Science, vol. 347, pp. 519–522, 1 2015.
  * [3] A. Kojima, K. Teshima, Y. Shirai, and T. Miyasaka, “Organometal halide perovskites as visible-light sensitizers for photovoltaic cells,” Journal of the american chemical society, vol. 131, no. 17, pp. 6050–6051, 2009.
  * [4] NREL, “Best research-cell efficiency chart,” U.S. Department of Energy, Office of Energy Efficiency and Renewable Energy, 2024 https://www.nrel.gov/pv/interactive-cell-efficiency.html.
  * [5] M. G. Ju, M. Chen, Y. Zhou, H. F. Garces, J. Dai, L. Ma, N. P. Padture, and X. C. Zeng, “Earth-abundant nontoxic titanium(iv)-based vacancy-ordered double perovskite halides with tunable 1.0 to 1.8 ev bandgaps for photovoltaic applications,” ACS Energy Letters, vol. 3, pp. 297–304, 2 2018.
  * [6] X. Zhang, L. Li, Z. Sun, and J. Luo, “Rational chemical doping of metal halide perovskites,” Chemical Society Reviews, vol. 48, pp. 517–539, 1 2019.
  * [7] G. Volonakis, A. A. Haghighirad, R. L. Milot, W. H. Sio, M. R. Filip, B. Wenger, M. B. Johnston, L. M. Herz, H. J. Snaith, and F. Giustino, “Cs2inagcl6: A new lead-free halide double perovskite with direct band gap,” Journal of Physical Chemistry Letters, vol. 8, pp. 772–778, 2 2017.
  * [8] T. T. Tran, J. R. Panella, J. R. Chamorro, J. R. Morey, and T. M. McQueen, “Designing indirect–direct bandgap transitions in double perovskites,” Materials Horizons, vol. 4, pp. 688–693, 7 2017.
  * [9] T. Ibn-Mohammed, S. C. Koh, I. M. Reaney, A. Acquaye, G. Schileo, K. B. Mustapha, and R. Greenough, “Perovskite solar cells: An integrated hybrid lifecycle assessment and review in comparison with other photovoltaic technologies,” Renewable and Sustainable Energy Reviews, vol. 80, pp. 1321–1344, 12 2017.
  * [10] M. Chen, M. G. Ju, A. D. Carl, Y. Zong, R. L. Grimm, J. Gu, X. C. Zeng, Y. Zhou, and N. P. Padture, “Cesium titanium(iv) bromide thin films based stable lead-free perovskite solar cells,” Joule, vol. 2, pp. 558–570, 3 2018.
  * [11] W. Li, S. Zhu, Y. Zhao, and Y. Qiu, “Structure, electronic and optical properties of cs2ti (br1-xyx) 6 (y= cl, i; x= 0, 0.25, 0.5, 0.75, 1) perovskites: the first principles investigations,” Journal of Solid State Chemistry, vol. 284, p. 121213, 2020.
  * [12] C. J. Bartel, C. Sutton, B. R. Goldsmith, R. Ouyang, C. B. Musgrave, L. M. Ghiringhelli, and M. Scheffler, “New tolerance factor to predict the stability of perovskite oxides and halides,” Science Advances, vol. 5, 1 2018.
  * [13] T. Islam, R. Jani, S. M. Al Amin, K. M. Shorowordi, S. S. Nishat, A. Kabir, M. Taufique, S. Chowdhury, S. Banerjee, and S. Ahmed, “Simulation studies to quantify the impacts of point defects: an investigation of cs2agbibr6 perovskite solar devices utilizing zno and cu2o as the charge transport layers,” Computational Materials Science, vol. 184, p. 109865, 2020.
  * [14] F. Matebese, R. Taziwa, and D. Mutukwa, “Progress on the synthesis and application of cuscn inorganic hole transport material in perovskite solar cells,” Materials, vol. 11, no. 12, p. 2592, 2018.
  * [15] F. Jannat, S. Ahmed, and M. A. Alim, “Performance analysis of cesium formamidinium lead mixed halide based perovskite solar cell with moox as hole transport material via scaps-1d,” Optik, vol. 228, p. 166202, 2021.
  * [16] A. Hima and N. Lakhdar, “Enhancement of efficiency and stability of ch3nh3gei3 solar cells with cusbs2,” Optical Materials, vol. 99, p. 109607, 2020.
  * [17] S. Kohnehpoushi, P. Nazari, B. A. Nejand, and M. Eskandari, “Mos2: a two-dimensional hole-transporting material for high-efficiency, low-cost perovskite solar cells,” Nanotechnology, vol. 29, no. 20, p. 205201, 2018.