# TEL’M: Test and Evaluation of Language Models††thanks: This work was
partially supported by DARPA under HR001119C0075, AFRL’s Autonomous Capability
Team 3 (ACT3) and Juniper Networks. Approved for public release; distribution
is unlimited.

George Cybenko
Thayer School of Engineering
Dartmouth College
Hanover NH 03755 USA Joshua Ackerman
Department of Computer Science
Dartmouth College
Hanover NH 03755 USA Paul Lintilhac
Thayer School of Engineering
Dartmouth College
Hanover NH 03755 USA

##  1 Overview

Language Models have demonstrated remarkable capabilities on some tasks while
failing dramatically on others [16, 74, 47]. The situation has generated
considerable interest in understanding and comparing the capabilities of
various Language Models (LMs) but those efforts have been largely ad hoc with
results that are often little more than anecdotal. This is in stark contrast
with testing and evaluation processes used in healthcare [30, 60, 29], radar
signal processing [20] and other defense areas [23]. The need for more
rigorous test and evaluation of artificial intelligence technologies has been
identified as a major challenge [32] which arises more broadly in computer
science research [22].

In this paper, we describe Test and Evaluation of Language Models (TEL’M) as a
principled approach for assessing the value of current and future LMs focused
on high-value commercial, government and national security applications. We
believe that this methodology could be applied to other Artificial
Intelligence (AI) technologies as part of the larger goal of “industrializing”
AI.

Most existing work aimed at evaluating LMs has been specific to certain
classes of tasks and prompts with empirical performance results that are
typically based on artificially defined benchmarks and ungrounded in rigorous
science [35, 27, 76, 79, 71]. Little attention is given to rigorous
“Experimental Design.” Many such results are based on small samples of tasks
and experiments with only qualitative summaries of performance. Moreover,
testing on benchmarks that are not aligned with or derived from end-use
applications of an LM may not be meaningful to the end-user.

The Five Ingredients of TEL’M : 1. Identification of LM Tasks of Interest 2.
Identification of Task Properties of Interest 3. Identification of Task
Property Metrics 4. Design of Measurement Experiments 5. Execution and
Analysis of Experiments

Figure 1: Many existing language model evaluations fail to follow or document
these simple steps.

TEL’M consists of five ingredients (see Figure 1) that will be described in
detail below. While these ingredients appear straightforward, they can be
nuanced and often nontrivial in the context of evaluating LMs for certain
tasks and properties, especially with respect to measurement and analysis.

A simple example is provided in the Appendix to illustrate and ground the
concepts. The example is meant to be “soup to nuts” in the sense that
motivated readers can recreate the datasets, language model architecture,
training and experiments with minimal effort. More complex examples are being
documented for future publication.

The main contribution of this project is a novel taxonomy of language model
properties and a rigorous methodology for measurement and analysis of those LM
task properties.

After a brief review of background information, we describe the TEL’M
methodology and its ingredients in detail with the running example. Because
language models and AI in general are currently such active research areas, we
cannot keep up with all the recent contributions and apologize if we have
omitted relevant work in this paper.

##  2 Background

It is assumed that readers are already familiar with Language Models of
various flavors such as:

  * •

Transformer-based Language Models (currently the most promising and studied
LMs) [78];

  * •

Multimodal Foundation Models such as Blip-2 [48] and CLIP [61];

  * •

Auto-regressive Language Models [15, 51];

  * •

Recurrent Neural Network Language Models [75];

  * •

State space language models [40];

  * •

Hybrid Models [24]

as well as the current and proposed use cases and the various technologies
underlying them [65, 70, 1].

There is growing interest in LM performance and benchmarks [13, 46, 18, 72,
64, 80, 16, 74, 47] with recent acknowledgement that this is a hard problem
[53]. Many suggestions are proposed in the commercial literature [17] and a
large number of benchmark-based methods have surfaced (Big Bench [67], GLUE
Benchmark, SuperGLUE Benchmark, OpenAI Moderation API, MMLU, EleutherAI LM
Eval, OpenAI Evals Adversarial NLI, LIT, ParlAI, CoQA, LAMBADA, HellaSwag,
LogiQA, MultiNLI, SQUAD to name a few). A review of existing approaches
demonstrates that they are not quantitative or rigorous enough to past muster
with respect to accepted testing requirements [55, 3]. In particular, existing
use of benchmarks do not investigate the extent to which a benchmark can
predict or quantify certain properties on future prompts (that is, statistical
soundness of any conclusions) and do not identify factors affecting
performance dependence as would be possible with more rigorous experimental
design and test execution.

LMs can be black box, gray box or white box according to the visibility into
the architecture and training data used to create an LM (see Table 1). Remote
Black Box LMs typically throttle the number of prompts so sustained access for
testing could be difficult unless priority access to an API is given. For
example, ChatGPT limits users to a small number of free prompts but allows
unlimited prompts on its subscription option. Additionally, reproducability
may not be guaranteed because of randomness in the response generation and/or
continuous adaptation of the LM platform.

LM Type |  Description |  Examples |  Different Use Cases
---|---|---|---
White Box |  Open source code;
User training data;
User trained. |  Sagemaker [2];
nanoGPT [4]. |  Can we use internal representations
of facts and concepts to measure computational properties?
Gray Box |  Foundation models;
Visibility into architecture but not into pre-training. |  Llama [52];
BERT [45];
OpenAI Davinci [56];
BLIP-2 [48]
CLIP [61] |  How does pre-training affect mission performance?
Did fine-tuning overcome biases in the pre-trained model?
Black Box |  Fully remote LM;
Competitor LM. |  Baidu ERNIE [8];
Yandex [5];
ChatGPT [57]. |  How does LM $A$ compare with LM $B$?
What are architectural features of an LM?
Table 1:  There are several categories of LMs with varying possible use cases
and observability. In White Box LMs, questions might be focused on various
aspects of performance, trust and alignment. On the other hand, Black Box
models could be of interest for comparisons and reverse engineering of LMs’
structures.

###  2.1 Common Inadequacies Made by Many LM Testing and Evaluation Efforts

We have come across several types of problems arising in published work
purporting to test and evaluate language models. The inadequacies we have
found are enumerated in Table 2. We do not indict specific authors or works by
attributing possible inadequacies they may have made according to our reading
of the publications. We encourage readers to make their own judgements when
reviewing publications that purport to evaluate language models.

Inadequacy |  Description |  Possible Remedies
---|---|---
Training-Testing Semantic Mismatch |  An LM trained to perform Task A but is tested for performance on semantically difference Task B |  Align semantics of training and testing with tasks of interest
Training-Testing Distribution Mismatch |  An LM trained to perform a task is tested for performance on semantically similar test data but the test data is drawn from a different distribution than the training data |  Check and align test data with either expected end-use data distribution or training data distribution
Tested samples drawn from open source benchmarks |  Public benchmarks could be used for training a foundation model and/or fine-tuning a proprietary model so results may not represent intended use-case performance |  Use custom test data aligned with end use case and/or understand if any benchmarks were used in training and/or tuning
Tested on too few samples |  Not enough test cases used to obtain statistically significant results |  Use more samples based on statistical analyses
No confidence or error bounds for results |  Reporting results with no expression of statistical confidence or error bounds |  Apply well-known and documented statistical procedures to get bounds
Uninterpretable metrics |  Results do not align with meaningful or interpretable assessments of interest to an end user |  Align properties and metrics with intended domain
Quality of “ground truth” is suspect |  The outputs of an LM is are compared with “ground truth” whose quality is not established or reported |  Estimate and report the quality of ground truth used in testing
Experimental details not included |  Details of datasets, testing, evaluation or other data not reported |  Specify details of experiments sufficient for reproducability
Table 2: There are multiple ways in which the testing and evaluation of
language models can be faulted. This table shows several major inadequacies
with suggestions for possible remedies.

##  3 The TEL’M Methodology

Steps in the TEL’M methodology as listed in Figure 1 will be described in
detail below and with concrete examples based on a simple paritycase-study
example in the Appendix. To refresh and prepare the reader we list the steps
again.

  1. 1.

Identification of LM Tasks of Interest: The problem(s) that the LM is expected
to solve.

  2. 2.

Identification of Task Properties of Interest: The properties that tasks (that
is, problem solutions) are relevant to test and quantify.

  3. 3.

Identification of Property Metrics: The manner in which the properties’
metrics are quantified and measured.

  4. 4.

Design of Measurement Experiments: The design of experiments that are to be
conducted to estimate the metrics and the statistical methodology used to
analyze results.

  5. 5.

Execution and Analysis of Experiments: The actual execution of the experiments
and subsequent analyses of results.

We recognize that following this methodology sequentially will be aspirational
in many LM test and evaluation situations because, in spite of best
intentions, unforeseen experimental and analysis issues can arise.
Nonetheless, people who conduct LM test and evaluation as well as consumers of
such results should be aware of experimental science pitfalls such
as“$p$-hacking” [42].

###  3.1 LM Tasks

LMs have demonstrated abilities on a wide range of tasks, from concrete and
relatively objective, such as authoring computer programs [28], to highly
subjective, such as authoring creative prose or poetry [31]. While it is
technically interesting to ask if an LM trained on a generic corpus of English
language can effectively respond to arithmetic [77] or planning [66] prompts,
in general we assume that most end-user concerns involve LMs that are trained
to perform specific kinds of domain-specific tasks and that rigorous test and
evaluation of those LMs will be done on samples of the same kinds of tasks.

Accordingly, it is standard practice to consider LM-based tasks that are
consistent with the design and training of an LM, with the general exception
of deliberately testing out-of-distribution (OOD) performance on tasks for
which the LM was not specifically trained. With this understanding, example
tasks that would be appropriate for test and evaluation are, for instance:

  * •

Example: TASK A: Use text and image prompts in a multimodal language model to
answer questions about the image, to caption images semantically and/or
synthesize novel properties of the image;

  * •

Example: TASK B: Synthesize programs that implement APIs for a complex cloud
computation whose components are user or third party sourced;

  * •

Example: TASK C: Integrate symbolic reasoning with neural processing to answer
natural language prompts that produce quantitative and logically consistent
military Courses of Action that are presented in a predefined, structured
Document Definition Language (DDL).

Ideally, Tasks of Interest should be defined well enough to anticipate the
properties one might ask of an LM performing that task. A Task of Interest
should have at least these characteristics:

  * •

Concreteness: The Task of Interest should be describable at a high semantic
level in prose for formulating hypotheses about the LM’s performance. Examples
of desired performance should be documented and explained clearly. For
example, the tasks of “knowing a lot of facts” and “doing my job” are not
concrete enough for meaningful testing;

  * •

Consistency: The task should be consistent with the design, training data and
actual training of the LM, although as noted above, it can be reasonable to
explore those boundaries. For example, the task “Play Miles Davis’s Kind of
Blue” cannot be done by ChatGPT because of copyright issues and the LM’s
limited output modality;

  * •

Observability: The task should be one for which an end user will be able to
quantitatively or qualitatively assess at least some meaningful properties of
LM responses that can be used for assessing task performance. In case the task
is related to inferring some aspect of an LM’s architecture or training data,
a property, for example the number of attention layers, should be observable
in the sense of classical control theory [50]; Examples of such properties are
whether the model is auto-regressive or uses causal masking. The observability
of a property may be LM specific because vendors of commercial LMs may not
want certain details of the implementation observable and might add barriers
or obfuscation techniques.

Part of the current excitement about LMs is the exploration of tasks which
they are either capable or incapable of doing so this will likely be a dynamic
area for some time, as the technology and expectations evolve.

###  3.2 Task Properties

Language Models operate by accepting an input prompt, which can be multimodal,
and producing a response, which can also be multimodal. More formally, and to
establish our notation, we regard LMs to be functions mapping an input prompt
space, $\cal{X}$, to an output response space, $\cal{Y}$. We will denote a
specific, fixed LM by $\cal{L}$ so that $\cal{L}:X\rightarrow Y$ and ${\cal
L}(x)=y$ is the response to prompt $x$.

In many cases, the output of an LM depends on various probabilistic
ingredients (such as sampling disciplines that depend on random number
generators that typically are not controllable by end users) so that ${\cal
L}(x)$ is possibly random and not always reproducible. That is, repeatedly
prompting an LM with the same prompt may not always produce the same response
as output. We will revisit this aspect of LMs once we start discussing
property metrics below.

By their very nature, different LM tasks will have have different properties
suitable for test and evaluation. Example properties of LM tasks are:

  * •

Accuracy: Accuracy of an LM, and any AI in general, is often a desirable
property. In classical information retrieval, there are several ways to define
accuracy [21]. On the other hand, if a task for an LM involves creativity,
alignment with facts may not be important. This type of property is discussed
in more detail in the metrics subsection.

  * •

Sensitivity: Task sensitivity captures the extent to which perturbations in
the inputs lead to changes in the outputs. This could be sensitivity to bit
flips in Boolean analysis, or a definition of sensitivity more appropriate for
natural language tasks. Task sensitivity has been found to be a good predictor
of task performance in both formal languages and natural languages [41].

  * •

Monotonicity: This property captures whether the performance of the LM is
monotonic in the complexity of the task quantified in some objective way [37].
For example, does the LM perform better on shorter prompts?

  * •

Training Efficiency: Significant progress is being made on the training of
foundational and fine-tuned LMs [58]. This property has to do with the
effectiveness and efficiency of such training with respect to training and
fine-tuning.

  * •

Prompt Efficiency: So-called “prompt engineering” sheds light on the extent to
which an LM’s ability to effectively perform a task can depend on the
structure of a prompt. Chain-of-thought (COT) prompting is an example of this
[73]. This task property has to do with the extent to which the LM end users
have to craft specific types of prompts with specific structures to achieve
effective responses.

  * •

Explainability: This is the extent to which responses of an LM on a task can
be interpreted and understood by human users. This property has already been
explored in detail for some AI technologies [44].

  * •

Extensibility: There is growing interest in how well fine-tuning will fare
when extending a foundation model to a specific use case and domain [12]. It
has to do with the size and scope of the fine-tuning used and the subsequent
performance improvements.

  * •

Creativity: “Hallucination” is often criticized as an undesirable LM artifact
but deviation from factual accuracy and prior training is a distinguishing
property of creativity [31]. In some tasks, a user might assign value to the
novelty of an LM response. In the national security domain, this could arise
when novel course of action plans are being sought for a specific tactical
situation. Recent work has explored internal properties of LMs for
distinguishing between fact and fiction [7].

  * •

Ethics and Biases: Ethics and biases are of great concern within the US
government [54] and can be factors across virtually all LM applications. While
the concerns are well-placed and appropriate, there are few principled
proposals for how to quantify and measure these properties. TEL’M can
investigate these issues and offer approaches for developing objective metrics
for such properties.

  * •

Usefulness: For some task products, such as LM authored code which may not be
accurate or correct, the performance of the LM might still be considered
“useful” in the sense that the code artifact is the basis for an effective
program construct with additional human effort. For example, an LM produced
code could have a high level structure that is initially not correct or
executable but provides a template that a human programmer can easily fix
[63].

We will denote by ${\cal F}$ the set of all functions mapping ${\cal X}$ to
${\cal Y}$ of which a trained LM ${\cal L}\in{\cal F}$ is but one example. For
properties, $P$, such as listed above, the subset of functions having the
property will be denoted by ${\cal F}_{P}$ so that ${\cal L}\in{\cal F}_{P}$
means that ${\cal L}$ has property $P$.

This is a very general framework for discussing properties. For example,
consider the “correctness” property. Assuming there is an objective way to
decide whether an individual LM response is correct or not, it may be too
strict a requirement that the LM is “always correct.” Instead, in many use
cases, it is acceptable to be correct “most of the time” in which case the
relaxed correctness property $P$ is that the LM produces a correct response,
say at least 95%, of the time a random prompt is input. This itself is a
property and ${\cal F}_{P}$ becomes the set of functions that are correct 95%
or more of the time. This is discussed in more detail below.

Figure 2: Hoeffding’s Inequality [43] can be used to bound the number of
samples needed to get confidence $1-\alpha$ for the probability that a
property, $P=P[\mu-\epsilon,\mu+\epsilon]$ as described in the Confidence
Intervals (CI) box. For example, to get an estimate of that probability to
within $\epsilon=0.1$ with confidence $1-\alpha=0.95$, we need at least 738
samples. Constraints and more sophisticated models for the property could
significantly reduce that number.

###  3.3 Property Metrics

The role of property metrics is to provide estimates of how “close” an LM,
${\cal L}$, is to having a property, $P$. To make this more rigorous, we
explicitly define several ingredients fundamental to TEL’M notions of
properties and metrics, along with the associated notation used here.

  * •

$D$ is a probability distribution over inputs as determined by the particular
use-case and task envisioned for the LM. This distribution could be formal and
explicit (such as the uniform distribution over binary strings of length 100)
or it could be entirely empirical (such as the set of prompts presented to
ChatGPT in the year 2023).

  * •

$TD_{N}$ is a “test data set” of size $N$, namely

$TD_{N}=\\{(x_{i},y_{i})\leavevmode\nobreak\ |\leavevmode\nobreak\
x_{i}\in{\cal X},{\cal F}(x_{i})=y_{i}\in{\cal Y},1\leq i\leq
N\leavevmode\nobreak\ \\}.$

In general, it is assumed that the $x_{i}$ are sampled independently from $D$,
specifically that $\leavevmode\nobreak\ x_{i}\sim D$. However, in cases where
the LM responses are not deterministic, it may be desirable and necessary to
repeatedly apply ${\cal L}$ to the same random input $x_{i}$ to get a sample
of the possible outputs for the same input prompt. In other test situations,
samples can be selected based on previous samples and their responses which
are then now independent samples. Such sampling is sometimes useful but beyond
the scope of this short overview.

  * •

${\cal A}$ is an algorithm or procedure that is applied to $TD_{N}$ to provide
an estimate of the extent to which the LM has the property $P$.

  * •

${\cal C}$ is statement about the “confidence” in the estimate that ${\cal A}$
will produce a similar estimate when presented with other test data sets,
$TD^{\prime}_{N}$. Such confidences are typically expressed in terms of the
distribution $D$ imposed on the test data. That is, a confidence of 0.95 means
that 95% of the time a similar test data set of size $N$ is obtained, a
similar estimate will be derived by the algorithm ${\cal A}$.

To see how these ingredients fit together, consider an LM, ${\cal L}$, and a
task for which “accuracy” is the property of interest, $P$. This assumes we
have knowledge of the correct response for input prompts, $x$, so that $T(x)$
are the correct responses for input $x$. We select $N$ (the value of which
will be discussed shortly) and obtain $N$ samples from $D$, $\\{(x_{i})\\}$
where $1\leq i\leq N$, and evaluate ${\cal L}(x_{i})$ to obtain the test data
results, $TD_{N}=\\{(x_{i},{\cal L}(x_{i})\\}$. Comparing ${\cal L}(x_{i})$
with the correct responses $T(x)$ defines a Boolean variable, $B(x)=1$ if
${\cal L}(x)=T(x)$ and $B(x)=0$ otherwise.

The expected value or mean of $B(x)$ over $D$ with respect to the underlying
probability distribution of input prompts in $D$, is

$\mu=\mathbb{E}_{x\sim D}(B(x))=\Pr\\{x\in
D|B(x)=1\\}=\lim_{N\rightarrow\infty}\frac{\sum B(x_{i})}{N}$ (1)

where the mean is the limit shown on the right by the Law of Large Numbers
(LLN) (note that the mean and variance of the random variable $B$ are both
finite). This means that the probability of the model ${\cal L}$ being correct
for a randomly chosen input prompt, $x$, is $\mu$ and can be approximated by
simply computing the fraction of tests that are correct. This is what the vast
majority of LM correctness testing efforts do now.

The rate at which the limit on the right in (1) approaches $\mu$ as
$N\rightarrow\infty$ can be estimated using one of several of probability
inequalities such as Chebyshev’s, Chernoff’s or Hoeffding’s [43]. In
particular, Hoeffding’s inequality states that for $0\leq B(x)\leq 1$ (which
is the case here for Boolean $B$)

$\Pr_{x_{i}\sim D}\left(\left|\frac{\sum
B(x_{i})}{N}-\mu\right|\leq\epsilon\right)\geq
1-2e^{-2N\epsilon^{2}}=1-\alpha$ (2)

which is a rigorous relationship between $\epsilon$, $\alpha$ and $N$. Namely,
selecting any two of these variable implies a bound on the third.

In terms of the ingredients listed above, we have

  * •

The test input data is drawn independently from $D$.

  * •

$TD_{N}$ is a “test data set” of size $N$ that is used to infer aspects (the
probability of being correct in particular).

  * •

The algorithm, ${\cal A}$, is the comparison of the LM responses with “truth”
and computation of the ratio between the number correct responses to the total
number of inputs tested. The $\mu$ and $\epsilon$ in inequality (2) is the
extent to which ${\cal L}$ is correct. In particular, the true but unknown
probability that ${\cal L}(x)$ is correct on a random input prompt, $x\sim D$,
is the mean value of $B(x)$ over $D$, namely $\mu$, which is approximated to
within a tolerance $\epsilon$ by the test sample average, $\sum B(x_{i})/N$.

  * •

The statement about the “confidence” in the estimate, ${\cal C}$, is the
assertion that a similar estimate when presented with other test data sets,
$TD^{\prime}_{N}$, will occur with probability at least $1-\alpha$.

Solving for $N$ in (2), we must have

$N\geq\frac{\ln(2/\alpha)}{2\epsilon^{2}}$

for the inequality to hold. A graphical depiction of the relationship between
$\alpha$, $\epsilon$ and $N$ is shown in Figure 2.

Once an appropriate number of test samples have been generated, there are
several classes of algorithms that could be applied to the test data for
property evaluation. We outline three main techniques below.

Confidence Intervals (CI): When tests are used to estimate scalar parameters
of an LM, such as accuracy or other properties that can be formulated as
averages, some form of confidence interval estimation can be used. For
example, accuracy can be quantified as the ratio of correct responses to all
responses. In general, if a property can be expressed as the average of
measurements of the test data (what we call a “simple property” as discussed
below) and no assumptions are made about that average value, then confidence
interval techniques can be applied. Using Law of Large Number type results
such as Hoeffding’s Inequality [43], it is possible to obtain statements such
as $\Pr({\cal L}\in{\cal F}_{P[\mu-\epsilon,\mu+\epsilon]})\geq 1-\alpha$ (3)
where $P[\mu-\epsilon,\mu+\epsilon]$ is the property that that the LM has
accuracy in the interval $[\mu-\epsilon,\mu+\epsilon]\subset[0,1]$. Given an
$\epsilon$ and $\alpha$, Hoeffding’s Inequality can be used to estimate how
large $N$, the test data size, should be. A graphical depiction of the
relationships between $\alpha$, $\epsilon$ and $N$ is shown in Figure 2. It is
important to note that the method of confidence intervals is relevant when no
prior assumptions about the property being tested are made except that the
test samples are randomly drawn and when a probability distribution of the
test statistic can be estimated.

Classical Hypothesis Testing (CHT): Unlike the method of confidence intervals,
hypothesis testing begins with a hypothesis about some parameters of the
models being tested. For example, we may want to know if an LM, say ${\cal L}$
with accuracy $\mu$, is more accurate than another LM or system for performing
the same task, say ${\cal L^{\prime}}$. If ${\cal L^{\prime}}$ has known
accuracy $\mu^{\prime}$, then we are asking if the accuracy, $\mu$, of ${\cal
L}$ satisfies $\mu\geq\mu^{\prime}$ which is a “hypothesis” about $\mu$ and
this question becomes the hypothesis. Classical hypothesis testing has been a
standard approach for analyzing hypotheses about systems based on experimental
data for over a century [20]. We will not review those concepts here and
instead point readers to review articles focused on the application of CHT to
machine learning which would be most relevant to this paper [26]. It is
important to note that CHT is relevant when some assumptions about the
property being tested are made and when a probability distribution of the test
statistic can be estimated or postulated. In spite of CHT’s long and
established history, there are ongoing critiques of CHT [39].

Property Testing (PT): Another paradigm has emerged in the past few years for
evaluating hypotheses about properties of functions. Called “Property
Testing”, this approach was developed to handle situations for which
confidence interval, classical hypothesis testing and other classical
statistical methods that require knowledge of a test statistic’s probability
distribution may not be easily computed and therefore cannot be directly
applied [38, 36]. As above, define $D$ is the probability distribution over
which samples of the task will be drawn and $P$ is the property being tested.
We say that the model function ${\cal L}$ is $\epsilon-$far from having
property $P$ when $\min_{h\in{\cal F}_{P}}\left\\{\Pr_{x\sim D}(h(x)\neq{\cal
L}(x))\right\\}\geq\epsilon.$ In other words, ${\cal L}$ is $\epsilon$-far
from ${\cal F}_{P}$ means at least an $\epsilon$ fraction of ${\cal L}$’s
responses would need to be changed in order to match the closest $h\in{\cal
F}_{P}$, that is, to have the property. As depicted in Figure 3 the closest
function with the desired property may or may not be in the language model
architecture class under consideration. Within this framework, an algorithm
${\cal PT}$ is called a “property tester” for property $P$ if, given an
$\epsilon\geq 0$ and a test data set $TD_{N(\epsilon)}$ of size $N(\epsilon)$
(that is, the test data size depends on $\epsilon$), it satisfies these two
conditions: • If ${\cal F}\in{\cal F}_{P}$ then
$\Pr_{TD_{N(\epsilon)}}\left\\{{\cal PT}\leavevmode\nobreak\ \text{applied to
}\leavevmode\nobreak\ TD_{N(\epsilon)}\text{ will conclude that }{\cal
F}\in{\cal F}_{P}\leavevmode\nobreak\ \leavevmode\nobreak\
\text{correctly}\right\\}\geq\frac{2}{3};$ • If ${\cal F}$ is $\epsilon$-far
from ${\cal F}_{P}$ then $\Pr_{TD_{N(\epsilon)}}\left\\{{\cal
PT}\leavevmode\nobreak\ \text{applied to }\leavevmode\nobreak\
TD_{N(\epsilon)}\text{ will conclude that }{\cal F}\notin{\cal
F}_{P}\leavevmode\nobreak\ \leavevmode\nobreak\
\text{correctly}\right\\}\geq\frac{2}{3};$ An effective property testing
algorithm requires that $TD(\epsilon)$ is reasonable and that the run time
complexity of the testing algorithm ${\cal PT}$ is “efficient.” It is
important to note that the property testing paradigm is silent about the
situation in which the model, ${\cal L}$, is neither in ${\cal F}_{P}$ nor
$\epsilon$-far from ${\cal F}_{P}$ which might be a large part of the
potential model space. We refer readers to recent publications that develop
the concepts and recent results in more detail [36].

Figure 3: Ideally, a property metric would provide some insight into how far
the LM under test is from having the property. For example, saying that a
system is 90% accurate should translate into meaning that 10% of the responses
would have to be changed to make the system 100% accurate. As discussed in the
text, this can be translated rigorously into meaning that the system is a
distance of 0.1 away from the property. However, the notion of “distance” from
an LM, ${\cal L}$ to the property under consideration has several nuanced
considerations as depicted in this figure. For example, the “closest” system
with the property may not be in the architecture class of language models
(such as transformers with the same specific sizes and configurations of
attention and feedforward layers as the LM under test, not just the generic
class of all transformers) to which the LM under test belongs. The problem of
determining whether any model in that same architectures class has the
property is typically a difficult theory oriented question about the
representational powers and trainability of language model architectures.

####  3.3.1 Simple Property Metrics

We call properties “simple” if the test data for the property consists of
individual independent prompt-response samples which are individually scored
and then averaged to obtain the property metric. Hoeffding’s or similar
probabilistic inequalities can be used to estimate the required test sample
sizes to obtain these property metrics with the required accuracy and
confidence, as discussed above in the Confidence Bounds description. We note
that any property of a task that is a limiting average of the some point-wise
property defined for each input is thus a “simple property” by definition.

All simple property metrics can be derived using confidence interval (CI)
estimation or classical hypothesis testing (CHT). Note that the failure rate
of the test can be regarded as the distance to some function having the
property. However, in the general context of language models, it is not known
in general if the closest function is in the same architectural class (for
example a transformer with specific hyperparameters) as the LM under test or
outside that class.

Figure 4: Simple task property metrics can be determined by independent
samples of an LM’s prompt-response behavior that are aggregated by averaging.
By contrast, compound task property metrics, such as monotonicity, require a
multiplicity of prompt-response pairs that are not simply averaged although
subsequent processing of averages may be needed to make inferences about a
compound property. Another dimension along which task property metrics vary is
their “order”. A first order property can be determined prima facie from the
prompt-response pairs by averaging. On the other hand, second and higher order
properties defer to other applications (possibly other LMs, parsers,
compilers, optimization packages, execution environments, etc. for example) to
estimate metric values. Those other applications themselves have to be tested
and evaluated with metrics that can be simple or compound as this figure
depicts. While 1st order property metrics are either simple or compound,
higher order property metrics can be combinations of both.

####  3.3.2 Compound Property Metrics

As defined above, simple property metrics are based on individual prompt-
response samples which are aggregated as averages allowing for the use of
Hoeffding’s or similar probabilistic inequalities. Compound properties on the
other hand are meaningful only in the context of multiple crafted prompt-
response samples by comparing and aggregating prompt-response samples for
certain sets of samples.

One such property is “monotonicity” of accuracy for example. Suppose we want
to test whether an LM’s accuracy, which is a simple property, decreases as
prompt complexity, quantified in some principled way, increases. If $\mu(n)$
is the simple property accuracy for prompts of length, $n$, then we are
interested in the whether the relation $\mu(n)\geq\mu(m)$ holds for all $m>n$.
While we can estimate whether $\mu(n)\geq\mu(m)$ using CHT, the monotonic
property at large requires combining multiple such tests using additional
probabilistic reasoning. In particular, if we determine that $\mu(n)\in[a,b]$
with probability at least $1-\delta_{1}$ and $\mu(m)\in[c,d]$ with probability
at least $1-\delta_{2}$ then we still need to estimate the probability that
$\mu(n)\geq\mu(m)$ given these measurements. Property testing for such
compound property metrics is an ongoing research area [37, 49].

Estimating an LM’s distance to monotonicity with respect to a property metric
can be done using a simple linear program as follows. Let the sequence of
sample averages of a simple property be
$\\{\widehat{\mu}_{1},...,\widehat{\mu}_{n}\\}$ where $\hat{\mu_{i}}$ is the
estimated property value for. Introduce slack variables
$\\{s_{1},...,s_{n}\\}$ and $\\{r_{1},...r_{n}\\}$, where $r_{i},s_{i}\geq 0$,
and find the minimum

$\epsilon=\min_{s_{i},r_{i}}\left[\sum_{i}\alpha_{i}(s_{i}+r_{i})\right]$

subject to the linear inequality constraints:
$\widehat{\mu}_{i}+s_{i}-r_{i}\geq\widehat{\mu}_{i+1}+s_{i+1}-r_{i+1}$ that
define monotonicity and $-\delta_{i}\leq s_{i}-r_{i}\leq\delta_{i}$ that
forces $\widehat{\mu}_{i}+s_{i}-r_{i}$ to be in the confidence interval for
the estimated $\widehat{\mu_{i}}$ as discussed above. Note that either
$r_{i}=0$ or $s_{i}=0$ or both must hold at the optimal solution. The
$\alpha_{i}$ are the fixed and known or assumed probabilities or fractions of
the input space that have complexities $i$ so that $\alpha_{i}\geq 0$ and
$\sum_{i}\alpha_{i}=1$. They are not variables in the linear program.

If this linear program has no solution, it shows that there are no values
within the multiple confidence intervals that result in a monotonic metric. On
the other hand, if there is a solution, then at the optimal solution,
$s_{i}-r_{i}$, is the minimum (weighted) amount that the estimated
$\widehat{\mu_{i}}$ needs to change in order for the property metric to become
monotonic. If the underlying property is simple, such as accuracy, then we can
interpret $s_{i}-r_{i}$ as the fraction of responses for complexity $i$
prompts that have to be changed for monotonicity to hold and with the
weighting $\alpha_{i}$, $\sum_{i}\alpha_{i}(s_{i}+r_{i})$ is the overall
fraction of responses across all complexities.

Another compound property of increasing interest is the sensitivity of a
language models’ responses to small perturbations in prompts [41]. Testing for
sensitivity requires sampling a neighborhood of a base-point prompt and
processing the associated responses to gauge the variability of those
responses with respect to the base-point.

####  3.3.3 Higher-Order Property Metrics

In some LM use cases, the evaluation of LM responses to estimate property
metrics requires nontrivial post-processing itself. For example, if we are
interested in LM accuracy, we need a way to estimate ground truth – that is,
what is “correct.”

If the task is a basic arithmetic problem, such as answering “What is the sum
of 25 and 48?”, we can easily have high confidence in the computed, objective
correct answer and so can score the response easily. On the other hand, if the
task involves natural language prompts and responses corresponding to a
question answering task [62] for example, then natural language responses have
to be evaluated which means that some confidence in the accuracy of the ground
truth is needed, as well as any automated comparison with a natural language
response and that ground truth. This has typically been done by using a
heuristic score such as BLEU [59] or by human scoring of natural language
responses (which may not be scalable to large, novel test samples).

As another example, consider the property of writing “correct” computer code
based on natural language descriptions of the desired code. Testing this
property requires checking the program against ground truth which begs the
question. For example, suppose we ask an LM to write a program to solve
certain types of planning problems [46]. Programs for solving those types of
problems might only produce approximate solutions because some aspect of the
problem involves solving NP-Hard optimization problems. In this case, we want
to have a metric on how close the response solutions are to computing optimal
solutions on instances of the planning problem. Accordingly, we would want to
quantify the LM performance on this task by asking if the LM produces planning
solutions that come to within say 90% of the optimal solution 95% of the time
it is presented a problem instance, and that the LM succeeds in doing this 80%
of the time.

For example, consider a task for which a model is compared with human
responses, attaining $p$ accuracy meaning that approximately a proportion of
$p$ responses are scored as correct and so $1-p$ are scored as incorrect with
respect to human responses. However, human responses are not in general
perfect [33, 25, 9]. To explore the implications of this, we introduce these
concepts:

  * •

The Human Accuracy is the accuracy, $q$, of the humans performing the task as
measured by comparison with some “truth” which is typically unknown but might
be inferred from other sources (although this is sometimes measured [9]);

  * •

The Test Accuracy of the model is the accuracy, $p$, of the model measured by
comparison with the test data of which only $q$ fraction is actually correct
and;

  * •

The True Accuracy, $r$, of the model is the accuracy of the model measured
against the unknown true results.

Figure 5: This illustrates the concepts and derivations surrounding various
possibilities for actual accuracy, $r$, of an LM in a higher-order testing
scenario when the data being tested against has true accuracy $q$. In this
context, when both the human and the model, which is scored again the human,
are “wrong” the model response is actually correct. For example, if $p=0.9$ is
the estimated model accuracy measured against data that has accuracy $q=0.95$
then the true value, $r$, will satisfy $0.9+0.95-1=0.85\leq r\leq
0.9+(1-0.95)=0.95$. Under the independence assumption for model and human
errors, $r=0.9\cdot 0.95=0.855$ which is almost at the lower bound for $r$.

If the correctness of human and LM responses to individual prompts are
independent random variables, then $r=pq$. However, if they are not
independent, then it is possible that most of the model test errors relative
to human true errors are aligned. That is, a model response is scored as
incorrect relative to the human response but seeing as the human response is
wrong, the model response is actually correct. So in an extreme case, if
$1-p\geq 1-q$ then it is possible that a fraction $1-q$ of model responses
that are scored in testing as wrong could in fact be correct because the human
generated test responses are wrong on those samples. In this extreme case, a
$p+(1-q)$ fraction of the model’s responses would be correct.

By similar reasoning, if $1-p\leq 1-q$, then all wrong model responses are on
samples for which the human responses are also wrong so we would have
$r=(1-p)+q$.

Finally, the other extreme case is when the wrong answers of the model are
disjoint from the wrong human answers and we get $r=1-(1-p)-(1-q)=p+q-1$. Note
we assumed that $0.5\leq p$ and $0.5\leq q$ in this binary classification
problem which can be obtained by flipping the response categories if needed.

Putting these together we get

$p+q-1=p-(1-q)\leq r\leq 1-p+q=q+(1-p)\leavevmode\nobreak\
\text{if}\leavevmode\nobreak\ 1-p\leq 1-q$

and

$p+q-1=p-(1-q)\leq r\leq 1+p-q=p+(1-q)\leavevmode\nobreak\
\text{if}\leavevmode\nobreak\ 1-p\geq 1-q.$

(Note that $r=pq$ lies between these upper and lower bounds because

$1-p+q-pq=(1-p)(1+q)\geq 0\geq p+q-pq-1=-(1-p)(1-q)$

if $p\geq q$ with a similar derivation for the case $p\leq q$ but with the
left term being $(1-q)(1+p)$. ) Figure 5 illustrates these derivations.

Explainability, creativity and usefulness, among other properties, that are
very subjective. For test and evaluation at scale, these properties would
likely have to be evaluated using tools developed by third parties or human
experts whose accuracy would ideally be quantified accordingly.

In summary, a higher-order LM property metric depends on downstream property
metrics of responses that themselves have to be tested and evaluated. We
believe that the above derived intervals for properties such as accuracy can
be extended to $n^{th}$-order metrics as well.

##  4 Design of Measurement Experiments

We have described the concepts of an LM task, task properties and various
metrics and algorithms for quantifying the extent to which tasks have those
properties. Although the design of experiments is listed as the second to last
ingredient, it should in practice be considered at a high level at the very
beginning of a TEL’M implementation. When possible, some form of design of
experiments methodology should be applied [6, 69].

Historically, a significant driver for early experimental design work was
motivated by the need to systematically control factors that could be used to
explain observed results. Factors in LM experimental design could be the use
or not of chain-of-thought prompting, the complexity and length of a prompt
within the class of possible prompts for that task, the distribution of
prompts over the space of possible prompts, including in-distribution vs out-
of-distribution sampling, among many others.

Ideally, TEL’M ingredients and especially the experiments would be negotiated
among the end-users, the LM authors, provider or vendor and the TEL’M test and
evaluation team.

In summary, a goal of experimental design is to frame experiments so that the
variations in property metrics (the dependent variables) can be rigorously
understood and quantified in terms of variations in features of the prompts
(the independent variables) which can include the underlying LM platforms.

The designs should be documented and reported together with any subsequent
deviations from the initial designs. For example, we have come across LM
evaluations (to remain anonymous) that report evaluations based on public
benchmarks without reporting whether those benchmarks were part of a training
or tuning data set. Subsequent evaluations using test samples not from the
benchmarks (so out-of-distribution with respect to the training data) find
significantly different performance. That could be a result of “$p$-hacking”
[14] or chance of course, the reality known only to the report authors if not
reported.

##  5 Execution and Analysis of Experiments

Once experiments have been designed, they have to be executed and the
resulting data analyzed in a principled way suitable for documentation,
dissemination and if needed, reproduction. A critical component of LM research
and evaluation continues to be appropriate access to the computing, storage
and data required for training and testing. Commercial language model
developers use resources that are typically difficult for outside, especially
academic, evaluators to match or even approximately approach.

Prior to conducting the actual computational experiments, it is appropriate to
address and document the following language model characteristics in play
during execution:

  * •

Is the LM under test deterministic or stochastic. That is, do repeated
identical prompts produce the same outputs or are they stochastically
generated and therefore responses to a single prompt may not be reproducible
by the same or other testers.

  * •

Is there a “temperature” setting? What is it and can it be controlled by the
LM’s end-user?

  * •

Some LMs maintain a context of previous prompts. Does the LM under test have a
context, can it be cleared, set or observed during testing?

  * •

Is the LM static or still being trained? This is especially important in
remote Black Box testing of LMs whose training could be continuous and/or
based on response feedback mechanisms so that tests may not be reproducible
over time.

  * •

Has the LM under test been trained on any benchmarks being used in testing? In
a black or gray box setting, this could be tested by comparing benchmark
performance on bespoke, novel prompt-response samples.

We have provided a straw man form for reporting LM test and evaluations in the
Appendix.

##  6 Summary

Rigorous testing and evaluation has been standard practice in industries such
as aerospace, pharmaceuticals, food, automotive and consumer appliances to
name a few. While the Turing Test was proposed early in the AI lifecycle [34],
recent advances in and applications of AI, specifically language models and
related generative systems, significantly expand the scope, scale and
complexity needed to evaluate modern AI systems in real-world applications. We
have proposed an initial framework for conducting test and evaluation of
language models which we believe can mature to be more relevant and useful as
the field matures.

We consider this to be a work-in-progress with the expectation that feedback,
constructive criticism and implementations of this framework will constantly
improve the scope and quality of the ideas presented here.

##  7 Acknowledgments

This work was partially supported by DARPA under HR001119C0075, AFRL’s
Autonomous Capability Team 3 (ACT3) and Juniper Networks. We thank colleagues
for discussions, critiques and helpful suggestions.

## References

  * [1] Sabeen Ahmed, Ian E Nielsen, Aakash Tripathi, Shamoon Siddiqui, Ravi P Ramachandran, and Ghulam Rasool.  Transformers in time-series analysis: A tutorial.  Circuits, Systems, and Signal Processing, pages 1–34, 2023.
  * [2] Amazon Web Services.  https://aws.amazon.com/blogs/machine-learning/training-large-language-models-on-amazon-sagemaker-best-practices/.
  * [3] Paul Ammann and Jeff Offutt.  Introduction to software testing.  Cambridge University Press, 2016.
  * [4] Anrej Karpathy.  https://github.com/karpathy/nanoGPT.
  * [5] Anrej Karpathy.  https://github.com/yandex/YaLM-100B.
  * [6] Jiju Antony.  Design of experiments for engineers and scientists.  Elsevier, 2023.
  * [7] Amos Azaria and Tom Mitchell.  The internal state of an LLM knows when its lying.  arXiv preprint arXiv:2304.13734, 2023.
  * [8] Baidu.  https://yiyan.baidu.com/welcome.
  * [9] William E Barlow, Chen Chi, Patricia A Carney, Stephen H Taplin, Carl D’Orsi, Gary Cutter, R Edward Hendrick, and Joann G Elmore.  Accuracy of screening mammography interpretation by characteristics of radiologists.  Journal of the National Cancer Institute, 96(24):1840–1850, 2004\.
  * [10] Satwik Bhattamishra, Arkil Patel, Phil Blunsom, and Varun Kanade.  Understanding in-context learning in transformers and LLMs by learning to learn discrete functions.  arXiv preprint arXiv:2310.03016, 2023.
  * [11] Satwik Bhattamishra, Arkil Patel, Varun Kanade, and Phil Blunsom.  Simplicity bias in transformers and their ability to learn sparse boolean functions.  arXiv preprint arXiv:2211.12316, 2022.
  * [12] Rishi Bommasani et al.  The foundation model transparency index.  https://crfm.stanford.edu/fmti/fmti.pdf.
  * [13] Rishi Bommasani, Percy Liang, and Tony Lee.  Holistic evaluation of language models.  Annals of the New York Academy of Sciences, 2023.
  * [14] Abel Brodeur, Nikolai Cook, and Anthony Heyes.  Methods matter: P-hacking and publication bias in causal analysis in economics.  American Economic Review, 110(11):3634–3660, 2020.
  * [15] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in Neural Information Processing Systems, 33:1877–1901, 2020.
  * [16] Mikhail Burtsev, Martin Reeves, and Adam Job.  The working limitations of large language models.  MIT Sloan Management Review, 65(1):1–5, 2023.
  * [17] Burak Ceylen.  Large language model evaluation in 2023: 5 methods.  https://research.aimultiple.com/large-language-model-evaluation/, October 3, 2023.
  * [18] Chang, Yupeng, Wang, Xu, Jindong Wang, Yuan Wu, Kaijie Zhu, Chen, Hao, Yang, Linyi, Yi, Xiaoyuan, Cunxiang Wang, Wang Yidong, et al.  A survey on evaluation of large language models.  arXiv preprint arXiv:2307.03109, 2023.
  * [19] David Chiang and Peter Cholak.  Overcoming a theoretical limitation of self-attention.  arXiv preprint arXiv:2202.12172, 2022.
  * [20] Domenico Ciuonzo, Antonio De Maio, and P Salvo Rossi.  A systematic framework for composite hypothesis testing of independent Bernoulli trials.  IEEE Signal Processing Letters, 22(9):1249–1253, 2015.
  * [21] Paul Clough and Mark Sanderson.  Evaluating the performance of information retrieval systems using test collections.  Information Research, 18(2):18–2, 2013.
  * [22] Andy Cockburn, Pierre Dragicevic, Lonni Besançon, and Carl Gutwin.  Threats of a replication crisis in empirical computer science.  Communications of the ACM, 63(8):70–79, 2020.
  * [23] National Research Council.  Statistics, testing, and defense acquisition: New approaches and methodological improvements.  https://nap.nationalacademies.org/catalog/6037/statistics-testing-and-defense-acquisition-new-approaches-and-methodological-improvements, 1998\.
  * [24] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov.  Transformer-xl: Attentive language models beyond a fixed-length context.  arXiv preprint arXiv:1901.02860, 2019.
  * [25] Samuel Dodge and Lina Karam.  A study and comparison of human and deep learning recognition performance under visual distortions.  In 2017 26th international conference on computer communication and networks (ICCCN), pages 1–7. IEEE, 2017.
  * [26] Frank Emmert-Streib and Matthias Dehmer.  Understanding statistical hypothesis testing: The logic of statistical inference.  Machine Learning and Knowledge Extraction, 1(3):945–962, 2019.
  * [27] Edward Beeching et al.  Open LLM Leaderboard.  https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard, 2023.
  * [28] Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M Zhang.  Large language models for software engineering: Survey and open problems.  arXiv preprint arXiv:2310.03533, 2023.
  * [29] US Food and Drug Administration.  The drug development process.  https://www.fda.gov/patients/learn-about-drug-and-device-approvals/drug-development-process, 2018\.
  * [30] FDA Center for Devices and Radiological Health.  Statistical guidance on reporting results from studies evaluating diagnostic tests - guidance for industry and FDA staff.  https://www.fda.gov/media/71147/download, March 13, 2007.
  * [31] Giorgio Franceschelli and Mirco Musolesi.  On the creativity of large language models.  arXiv preprint arXiv:2304.00008, 2023.
  * [32] Laura Freeman.  Test and evaluation for artificial intelligence.  Insight, 23(1):27–30, 2020.
  * [33] Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey.  Experts, errors, and context: A large-scale study of human evaluation for machine translation.  Transactions of the Association for Computational Linguistics, 9:1460–1474, 2021.
  * [34] Robert M French.  The turing test: the first 50 years.  Trends in cognitive sciences, 4(3):115–122, 2000.
  * [35] Yao Fu, Litu Ou, Mingyu Chen, Yuhao Wan, Hao Peng, and Tushar Khot.  Chain-of-thought hub: A continuous effort to measure large language models’ reasoning performance.  arXiv preprint arXiv:2305.17306, 2023.
  * [36] Oded Goldreich.  Introduction to property testing.  Cambridge University Press, 2017.
  * [37] Oded Goldreich, Shafi Goldwasser, Eric Lehman, and Dana Ron.  Testing monotonicity.  In Proceedings of the 39th Annual Symposium on Foundations of Computer Science, FOCS ’98, page 426, USA, 1998. IEEE Computer Society.
  * [38] Oded Goldreich, Shafi Goldwasser, and Dana Ron.  Property testing and its connection to learning and approximation.  Journal of the ACM (JACM), 45(4):653–750, 1998.
  * [39] Sander Greenland, Stephen J Senn, Kenneth J Rothman, John B Carlin, Charles Poole, Steven N Goodman, and Douglas G Altman.  Statistical tests, p values, confidence intervals, and power: a guide to misinterpretations.  European journal of epidemiology, 31:337–350, 2016.
  * [40] Albert Gu and Tri Dao.  Mamba: Linear-time sequence modeling with selective state spaces.  arXiv preprint arXiv:2312.00752, 2023.
  * [41] Michael Hahn, Dan Jurafsky, and Richard Futrell.  Sensitivity as a complexity measure for sequence classification tasks.  arXiv preprint arXiv:2104.10343v1, 2021.
  * [42] Megan L Head, Luke Holman, Rob Lanfear, Andrew T Kahn, and Michael D Jennions.  The extent and consequences of p-hacking in science.  PLoS biology, 13(3):e1002106, 2015.
  * [43] Wassily Hoeffding.  Probability inequalities for sums of bounded random variables.  The collected works of Wassily Hoeffding, pages 409–426, 1994.
  * [44] Robert R Hoffman, Shane T Mueller, Gary Klein, and Jordan Litman.  Metrics for explainable AI: Challenges and prospects.  arXiv preprint arXiv:1812.04608, 2018.
  * [45] huggingface.  https://huggingface.co/bert-base-multilingual-cased.
  * [46] Susmit Jha, Sumit Kumar Jha, Patrick Lincoln, Nathaniel D Bastian, Alvaro Velasquez, and Sandeep Neema.  Dehallucinating large language models using formal methods guided iterative prompting.  In 2023 IEEE International Conference on Assured Autonomy (ICAA), pages 149–152. IEEE, 2023.
  * [47] Michael King.  Large language models are extremely bad at creating anagrams.  Authorea Preprints, 2023.
  * [48] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  arXiv preprint arXiv:2301.12597, 2023.
  * [49] Paul Lintilhac, Joshua Ackerman, and George Cybenko.  Testing and evaluating artificial intelligence applications.  to appear in IEEE LangSec 2024, 2024.
  * [50] Yang-Yu Liu, Jean-Jacques Slotine, and Albert-László Barabási.  Observability of complex systems.  Proceedings of the National Academy of Sciences, 110(7):2460–2465, 2013.
  * [51] Eran Malach.  Auto-regressive next-token predictors are universal learners.  arXiv preprint arXiv:2309.06979, 2023.
  * [52] Meta.  https://ai.meta.com/blog/large-language-model-llama-meta-ai/.
  * [53] Arvind Natarayan and Sayash Kapoor.  Evaluating LLMs is a minefield.  https://www.cs.princeton.edu/~arvindn/talks/evaluating_llms_minefield/, October 3, 2023.
  * [54] Office of Science and Technology Policy.  https://www.whitehouse.gov/ostp/ai-bill-of-rights/#discrimination, 2023\.
  * [55] Office of the Director, Operational Test and Evaluation (US DOD).  https://www.esd.whs.mil/Portals/54/Documents/DD/issuances/dodi/500089p.PDF.
  * [56] OpenAI.  https://platform.openai.com/docs/models/overview.
  * [57] OpenAI.  https://chat.openai.com/.
  * [58] OpenAI.  https://openai.com/research/ai-and-efficiency.
  * [59] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: a method for automatic evaluation of machine translation.  In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002.
  * [60] Alan Phillips and Vincent Haudiquet.  ICH E9 guideline - statistical principles for clinical trials: a case study.  Statistics in Medicine, 22(1):1–11, 2003.
  * [61] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [62] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang.  Squad: 100,000+ questions for machine comprehension of text.  In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, 2016.
  * [63] Veselin Raychev, Martin Vechev, and Eran Yahav.  Code completion with statistical language models.  In Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 419–428, 2014\.
  * [64] Burr Settles.  Active learning literature survey.  https://minds.wisconsin.edu/bitstream/handle/1793/60660/TR1648.pdf, 2009\.
  * [65] Alex Sherstinsky.  Fundamentals of recurrent neural network (rnn) and long short-term memory (lstm) network.  Physica D: Nonlinear Phenomena, 404:132306, 2020.
  * [66] Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg.  Progprompt: Generating situated robot task plans using large language models.  In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 11523–11530. IEEE, 2023.
  * [67] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al.  Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.  arXiv preprint arXiv:2206.04615, 2022.
  * [68] Lena Strobl, William Merrill, Gail Weiss, David Chiang, and Dana Angluin.  Transformers as recognizers of formal languages: A survey on expressivity.  arXiv preprint arXiv:2311.00208, 2023.
  * [69] Jacqueline K Telford.  A brief introduction to design of experiments.  Johns Hopkins APL Technical Digest, 27(3):224–232, 2007.
  * [70] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in Neural Information Processing Systems, 30, 2017\.
  * [71] Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, et al.  Pandalm: An automatic evaluation benchmark for LLM instruction tuning optimization.  arXiv preprint arXiv:2306.05087, 2023.
  * [72] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al.  Emergent abilities of large language models.  arXiv preprint arXiv:2206.07682, 2022.
  * [73] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.  Chain-of-thought prompting elicits reasoning in large language models.  Advances in Neural Information Processing Systems, 35:24824–24837, 2022.
  * [74] Yotam Wolf, Noam Wies, Yoav Levine, and Amnon Shashua.  Fundamental limitations of alignment in large language models.  arXiv preprint arXiv:2304.11082, 2023.
  * [75] Jianqiong Xiao and Zhiyong Zhou.  Research progress of rnn language model.  In 2020 IEEE International Conference on Artificial Intelligence and Computer Applications (ICAICA), pages 1285–1288. IEEE, 2020.
  * [76] Dingli Yu, Simran Kaur, Arushi Gupta, Jonah Brown-Cohen, Anirudh Goyal, and Sanjeev Arora.  Skill-mix: A flexible and expandable family of evaluations for ai models.  arXiv preprint arXiv:2310.17567, 2023.
  * [77] Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, and Songfang Huang.  How well do large language models perform in arithmetic tasks?  arXiv preprint arXiv:2304.02015, 2023.
  * [78] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al.  A survey of large language models.  arXiv preprint arXiv:2303.18223, 2023.
  * [79] Shen Zheng, Yuyu Zhang, Yijie Zhu, Chenguang Xi, Pengyang Gao, Xun Zhou, and Kevin Chen-Chuan Chang.  Gpt-fathom: Benchmarking large language models to decipher the evolutionary path towards gpt-4 and beyond.  arXiv preprint arXiv:2309.16583, 2023.
  * [80] Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al.  Representation engineering: A top-down approach to ai transparency.  arXiv preprint arXiv:2310.01405, 2023.

## Appendix - Illustration of TEL’M for Parity Problems

### Case-Study: Binary String Parity

Many recent efforts have investigated relationships between Boolean functions
(such as parity) and transformer architectures [10, 68, 11, 19] with a focus
on which functions can be effectively represented and learned by transformers.
This growing body of work provides a sound theoretical basis for this class of
problems, the simplest of which we explore here for pedagogical and
illustrative purposes. These experiments can be easily replicated by the
reader.

A few facts about parity are:

  * •

The parity of a finite binary string of any size is computed by adding the
bits in the string and then noting whether the sum is even or odd. If the sum
is even, the parity is 0 and if the sum is odd, the parity is 1;

  * •

The formal language defined by strings of even parity is a regular language
accepted by a simple two state deterministic finite automaton. As such it is
one of the simplest nontrivial regular languages;

  * •

Parity is extremely “sensitive” to perturbations in the input string because
flipping any single bit in a string changes its parity;

  * •

It has been shown that transformer-based language models have difficulty
learning the parity function for even moderately sized string lengths and
generalization to longer length strings is poor [10, 68, 11, 19];

  * •

Two reasonable properties to test for this task are accuracy (a simple metric)
and monotonicity (a compound metric);

  * •

Technically speaking, both metrics are also second-order properties because
they rely on an accurate computation of a string’s parity but because this is
done with a simple Python script that is easy to verify, we do not discuss
this aspect further.

For these reasons, we have selected parity as an example to illustrate TEL’M
reporting.

#### Training and Testing of Transformer-Based Parity Classifier

We trained and tested four parity classifiers based on the Chiang and Cholak
Transformer [19]111https://github.com/ndnlp/parity. Our goal is to illustrate
TEL’M suggested methodologies, terminology and reporting, not to study
language model solutions to the parity problem. The four models are:

  * •

two trained on strings of length 8-15, which we call the “Narrow Train”
classifier (test results are shown in Figures 7 and 9), and;

  * •

two trained on strings of length 8-45, which we call the “Wide Train”
classifier (test results are shown in Figures 6 and 8).

For each of these two groups, we performed testing using 1163 samples, as well
as a test with about 10 times as many samples, so 11628 test samples for each
length that we tested. We chose this approach to demonstrate the role of
increased test sample size in getting smaller confidence intervals and tighter
lower bounds on the distance to monotonicity.

Each of the four trained classifiers was tested on the parity task using
strings of length between 3 and 45, although we limited our visualizations as
well as our distance calculation to strings of length between 8 and 45. We did
this for two reasons: first, since all classifiers performed nearly perfectly
on very short strings, it does not affect the distance to monotonicity
calculation. Second, these short strings are going to be very highly
duplicated during testing, since we use over 1000 samples per string length
which larger than the number of short length strings. For the smaller sample
size test, we drew a total of 50,000 samples, averaging out to the 1163
samples per string length, while for the larger sample size test we drew
500,000 samples, or roughly 11628 samples per string length. During testing,
the length of the string was selected uniformly between 8 and 45, and then a
string was selected uniformly at random from all stirngs of that length.

The confidence interval derived from our tests as based on Hoeffding’s
Inequality, has a two-sided confidence level of $99.9\%$. Since there are
$45-8+1=38$ confidence intervals, one for each string length tested, all of
them hold simultaneously with probability $0.999^{38}=96.3\%$.

The four plots below show three confidence intervals for each of the 38 string
lengths. The original sample averages and their confidence intervals are shown
in green but many are occluded by red confidence intervals. When a green
confidence interval is not shown, it is beneath the red interval.

Our linear programming approach to finding the minimal distance of these
measurements to monotonicity are the shifted confidence intervals (shifted
using one of the slack variables in the solution to the linear program) shown
in red. Any string length that doesn’t show green signifies that the original
and shifted confidence intervals are completely overlapping, i.e. no shift was
necessary (in the best-case) in order to establish monotonicity with its
neighbors (and therefore overall). The distance between the green and red
intervals is the“cost” incurred at that string length, or the amount that the
entire interval would need to be shifted in order to contain a solution that
is overall monotonic. Finally, the blue dots represent the actual linear
program solutions, which is a point within the (possibly) shifted confidence
interval that achieves monotonicity.

Note that some of the plots indicate a solution that decreases seemingly
unnecessarily compared to its previous neighbor. The solution for the last
four string lengths (42,43,44,45) in Figure 6 could be shifted up slightly,
and as long as they are below the average at string length 41, the entire
function would still be monotonic. This is a reflection of the fact that the
solution to the linear programming problem is not unique. This non-uniqueness
does not affect the lower bound on the distance in any way, and the fact that
the plots show the lowest possible sample averages that attain the minimum
shows an arbitrary minimal solution, and an artifact of the lpSolve package in
R.

Figure 6:  This plot shows the calculations behind the lower bound on distance
to monotonic accuracy for the parity classifier trained on strings of length
8-45 (”wide train”). The total distance of this accuracy function from
monotonic is lower bounded by $0$, as there is a solution within each
confidence interval that is fully monotonic. The average value of $\epsilon$
in Hoeffding’s inequality (equal to $\frac{1}{2}$ the width of the confidence
interval for each string length, ignoring truncations at 0 and 1) is $0.057.$
As noted, the confidence that the true accuracy is in an interval is 99.9%.
Figure 7:  This plot shows the calculations behind the lower bound on distance
to monotonic accuracy for the parity classifier trained on strings of length
8-15 (”narrow train”). The total distance of this accuracy function from
monotonic is again lower bounded by $0.0254$. The average value of $\epsilon$
in Hoeffding’s inequality is $0.057.$ As noted, the confidence that the true
accuracy is in an interval is 99.9%. Figure 8:  This plot shows the
calculations behind the lower bound on distance to monotonic accuracy for the
parity classifier trained on strings of length 8-45 (”wide train”). The total
distance of this accuracy function from monotonic is lower bounded by $0$, as
there is a solution within each confidence interval that is fully monotonic.
The average value of $\epsilon$ in Hoeffding’s inequality is $0.018.$ As
noted, the confidence that the true accuracy is in an interval is 99.9%.
Figure 9:  This plot shows the calculations behind the lower bound on distance
to monotonic accuracy for the parity classifier trained on strings of length
8-15 (”narrow train”). The total distance of this accuracy function from
monotonic is lower bounded by $0.0368$. The average value of $\epsilon$ in
Hoeffding’s inequality is $0.018.$ As expected, the more samples we take given
the same desired confidence level, the narrower the confidence intervals, and
the more we have to “move around” the confidence intervals in order to obtain
a monotonic function. As noted, the confidence that the true accuracy is in an
interval is 99.9%.

TEL’M Template | Prepared by Paul Lintilhac<EMAIL_ADDRESS>April 8 2024
---|---
|  Name |  Chiang and Cholak Transformer (2 layers, 2 heads) [19] https://github.com/ndnlp/parity
Language Model |  Version |  GitHub, downloaded March 2024
|  Training details |  50,000 epochs, 100 steps each, batch size 1 (total of 5,000,000 samples with possible duplicates.) Trained on strings of length 8-15 (narrow train) and 8-45 (wide train)
|  Benchmarks Used in Training Data |  None.
|  Fine-tuning details |  None. Model trained from scratch.
|  (if any) |
|  Adaptations |  Using Classification Head on CLS (classify token) position.
|  (if any) |
Task tested |  Description |  Compute parity of a variable length binary string as in Chiang et. al. [19]
|  Dependencies |  Integer remainder from Python 3.9.6 library. (label = len([a for a in w if a == 1]) % 2 == 1 )
|  Description |  The model has the sole purpose of computing the parity of bit strings of variable length.
Property tested |  Number of Samples |  50,000 (larger interval) and 500,000 (smaller interval)
|  Distribution of Samples |  Uniform over length and uniform within strings of that length.
|  Testing Algorithm |  Lower bound on distance to monotonicity using a linear program.
|  Description |  We measure the distance to monotonicity under the distribution that selects a string length uniformly and then a uniform random string of that length.
Property Metric |  Type Used |  Compound
|  Distance Distribution |  Uniform over length and uniform within strings of that length.
|  Name & Location |  Mac M3 Pro Max in Shelburne, VT
Test Infrastructure |  Description |  Commercial laptop
|  Time used for testing |  Testing done in under 10 minutes.
|  Post processing |  Tests conducted in Python 3.9.6, Linear Program and plots implemented in R
|  Benchmarks |  None
|  used (if any) |
|  Stochasticity & Temperature |  None.
|  Open source model |  Github for Chiang and Cholak Transformer: https://github.com/ndnlp/parity
Reproducability |  Open source training data |  None. Training data randomly generated.
|  Open source testing data |  None. Testing data randomly generated.
Table 3: This is a draft TEL’M report of the tests described here. It is
proposed as a template for reporting test and evaluation results for general
language models and even Ai systems.