# AutoMix: Automatically Mixing Language Models

Aman Madaan♠ Pranjal Aggarwal♢ 11footnotemark: 1 Ankit Anand ‡ Srividya
Pranavi Potharaju †
Swaroop Mishra‡ Pei Zhou♣ Aditya Gupta Dheeraj Rajagopal† Karthik Kappaganthu†
Yiming Yang♠ Shyam Upadhyay† Mausam♢ Manaal Faruqui†
$\spadesuit$ Carnegie Mellon University $\dagger$ Google $\ddagger$ Google
DeepMind
$\diamondsuit$ IIT Delhi $\clubsuit$ University of Southern California
<EMAIL_ADDRESS>Equal Contribution. Work started and partly
done during Aman’s internship at Google.

###### Abstract

Large language models (LLMs) are now available in various sizes and
configurations from cloud API providers. While this diversity offers a broad
spectrum of choices, effectively leveraging the options to optimize
computational cost and performance remains challenging. In this work, we
present AutoMix, an approach that strategically routes queries to larger LMs,
based on the approximate correctness of outputs from a smaller LM. Central to
AutoMix is a few-shot self-verification mechanism, which estimates the
reliability of its own outputs without requiring training. Given that
verifications can be noisy, we employ a meta verifier in AutoMix to refine the
accuracy of these assessments. Our experiments using LLaMa2-13/70b, on five
context-grounded reasoning datasets demonstrate that AutoMix surpasses
established baselines, improving the incremental benefit per cost by up to
89%.111Code and data available at https://github.com/automix-llm/automix.

##  1 Introduction

Human problem-solving inherently follows a multi-step process: generate a
solution, verify its validity, and refine it further based on verification
outcomes. The emulation of this self-refinement and reflective behavior has
gained attention in the recent research (Pan et al., 2023; Madaan et al.,
2023; Reid and Neubig, 2022; Schick et al., 2022; Welleck et al., 2022; Shinn
et al., 2023). Classic self-refine paradigms consistently employ a single
model across all problem-solving stages, demonstrating effectiveness in
certain scenarios (Madaan et al., 2023; Shinn et al., 2023). Yet, the
intrinsic complexity and variability of tasks, from simplistic (e.g., binary
classification on separable data) to complex (e.g., code generation) and
potentially unsolvable (e.g., certain forms of multi-step reasoning), motivate
an alternative approach of model switching. Model switching iteratively
queries over models of disparate sizes and capabilities, verifying feedback at
each step and determining whether to accept the output or route to a more
capable, albeit computationally intensive, model (Liu et al., 2020; Zhou et
al., 2020; Madaan and Yang, 2022; Geng et al., 2021; Schuster et al., 2022).

Past studies in model-switching strategies predominantly rely on separate
models trained explicitly for each step or require access to logits(Chen et
al., 2023; Welleck et al., 2022; Reid and Neubig, 2022). However, modern LLM
often provide access solely through black-box APIs, restricting direct model
optimization and adaptability due to the unavailability of fine-tuning
capabilities and weight access. In response to this, we introduce AutoMix, a
method that utilizes black-box LLM APIs, circumventing the necessity for
separate models or logits access by adopting few-shot learning strategies
(Brown et al., 2020) and implementing self-verification. Our method proposes
strategies for each step of problem-solving: solution generation,
verification, and routing, all assuming we only have access to black-box LLMs.

In contrast to existing approaches, which generally classify tasks as Simple
or Complex for model routing, AutoMix integrates a third category of
Unsolvable queries. These queries are likely unsolvable even by a Large
Language Model (LLM) and should not be routed to larger models if identified
early. This consideration allows AutoMix to judiciously allocate computational
resources, preventing unwarranted computational spending on these particularly
challenging instances.

We use context-grounded few-shot entailment to evaluate the consistency of
generated answers with the provided context, without requiring a large amount
of human-labeled data (Poliak, 2020; Dagan et al., 2022). For example, an
answer discussing "desert animals" in a context focused on "aquatic life"
would be flagged as inconsistent. However, recognizing that self-verification
can sometimes be inconsistent or noisy (Huang et al., 2023), we introduce a
_meta-verifier_ to evaluate the reliability of the initial verification. The
meta-verifier acts as a secondary check, providing an additional layer of
confidence assessment to ensure that the decision to route a task to a larger
or smaller model is well-founded.

In summary, our contributions are:

  * $\bullet$

We introduce AutoMix, a method that strategically leverages black-box LLM APIs
for generating a solution, verifying the solution, and switching to a larger
language model, everything without access to model weights, gradients, or
logits.

  * $\bullet$

We also show that context-grounded entailment is a reasonable but noisy proxy
for self-verification. To deal with this noise, we propose a POMDP-based meta-
verification mechanism that helps improve the reliability of the final
decision.

  * $\bullet$

We propose and introduce the Incremental Benefit Per Unit Cost (ibc) metric, a
novel measure that quantifies the efficiency of integrating smaller and larger
language models.

  * $\bullet$

We present empirical evidence from experiments on five context-grounded
reasoning datasets using the language models llama2-13b and llama2-70b as the
small (SLM) and large (LLM) language models. Our results demonstrate that
AutoMix surpasses baselines, enhancing the incremental benefit per cost by up
to 89%.

Figure 1: AutoMix: Given a context (like an article) and a question $q$, an
initial answer (1890 AD) is generated with the smaller language model (SLM).
The answer is self-verified by the SLM, yielding a noisy verification score.
The Meta-Verifier subsequently assesses verifier’s results. Based on the meta-
verifier’s decision, either the initial answer (1890 AD) is returned, or the
question is rerouted to a larger language model (LLM) to enhance accuracy.

##  2 AutoMix: Few-shot Self-Verification and Meta-Verification

⬇

Context: {context}

\parQuestion: {question}

\parAI Generated Answer: {generated_answer}

\parInstruction: Your task is to evaluate if the AI Generated Answer is
correct, based on the provided context and question. Provide the judgement and
reasoning for each case. Choose between Correct or Incorrect.

\parEvaluation:"

List of codeblocks 1 Verification Prompt. The verification process is framed
as a natural language entailment task, where the model determines the validity
of the model-generated answer with respect to the context and question. We use
a generic few-shot prompt for all tasks (prompt in section D.1).

#### Task and setup

We tackle the problem of context-grounded question answering, where given a
context $\mathcal{C}$ (e.g., stories, newswire, or research article) and a
question $q$, the model is tasked with generating an accurate and coherent
answer, consistent with the provided context. Our choice of tasks is motivated
by two key concerns: (1) longer queries are more computationally demanding,
underscoring the need for an approach like AutoMix to navigate the cost-
accuracy trade-off, and (2) the context allows for cross-checking preliminary
answers with available information using self-verification (described
shortly), aiding in identifying inconsistencies as ungrounded is challenging
(Pan et al., 2023; Huang et al., 2023).

We deploy two distinct models: a smaller, cost-efficient model, denoted as
SLM, and a larger, more accurate yet costly model, LLM. Our objective is to
optimize performance while staying economical. An initial answer,
$\mathcal{A}_{s}$, is generated using the smaller SLM.

#### Few-shot Verification

To assess the trustworthiness of $\mathcal{A}_{s}$, we employ a few-shot
verifier, $\mathcal{V}$, which ascertains the validity of SLM’s outputs and
decides if a query should be redirected to LLM. Verification is framed as an
entailment task (Poliak, 2020; Dagan et al., 2022), aiming to determine if the
answer generated by SLM aligns with the provided context. Specifically, the
verifier gauges $v=\text{p}(\text{correct =
1}\mid\mathcal{A}_{s},\mathcal{C},q)$, with correct $=1$ indicating that
$\mathcal{A}_{s}$ is correct. The verification prompt is outlined in Figure 1.
We use the same verification prompt for all tasks. Figure 2 shows an example.

###  2.1 Meta-verifier

Given the potential inconsistency or noise in verifier outcomes, a secondary
evaluation mechanism, which we term the _meta-verifier_ , is crucial to vet
the verifier’s conclusions. In particular, the verifier is tasked with
determining whether the SLM’s answer is entailed by the context, and this
decision is made without considering the inherent difficulty of the problem.
Notably, routing Unsolvable queries to the LLM is resource-inefficient and
does not enhance performance. While ascertaining the ground truth of query
difficulty is non-trivial, verification probability and trends from historical
data inferred using validation set, can provide insightful guidance. Formally,
we define the meta-verifier’s outputs as
$m(v,\mathcal{A}_{s},\mathcal{C},q)\rightarrow\\{0,1\\}$, where $m=1$ implies
the verifier’s output can be trusted.

Addressing the notable challenges of self-correction in large language models
(Madaan et al., 2023; Huang et al., 2023), our method employs a non-LLM setup
for meta-verification to avoid escalating issues like hallucination and
reasoning errors (Dziri et al., 2023). The versatile meta-verifier can adopt
various advanced learning strategies, from supervised to reinforcement
learning, explored further in upcoming sections. Subsequent sections provide a
deeper exploration into two particular implementations of this strategy.

#### Thresholding

In this simplistic meta-verifier approach, the decision is made based on
probability of verifier being correct with a threshold t, defined as
$m_{t}(v)=0$ for $v<t$ and $m_{t}(v)=1$ for $t\geq v$. Intuitively, a high
probability of verifier indicates that verifier is confident in it’s decision
and can be trusted. For black-box language models, the probability of
correctness can be derived by sampling $k>1$ samples at a higher sampling
temperature.

Figure 2: Context-Grounded Self-Verification in Action. The example showcases
the verifier, utilizing the same model as the answer generator, identifying
and rejecting an inaccurate answer—He took it in 1990—by effectively
leveraging the context. The example uses llama2-13b for both generation and
verification on a coqa dataset instance.

#### Using a POMDP

In the context of meta-verifier, we observe that all the queries in this two
language model setup could be categorized in three different categories:
Simple, Complex, and Unsolvable. The simple queries are addressable by SLM
itself, the complex queries are addressable by LLM but not by SLM and
Unsolvable queries are so complex that they can’t be addressed by either LLM
or SLM. Hence, a ground truth oracle should route only the complex queries but
not unsolvable queries. Since the ground truth state, i.e category of query,
is not known and unobserved, we formulate this decision problem as a Partially
Observable Markov Decision Process (POMDP) Monahan (1982). POMDP presents a
robust framework, offering a structured way to manage and navigate through the
decision spaces where the system’s state is not fully observable. A POMDP is
defined by a tuple $(S,A,T,R,\Omega,O)$, where $S$ is a set of states, $A$ is
a set of actions, $T$ represents the state transition probabilities, $R$ is
the reward function, $\Omega$ is a set of observations, and $O$ is the
observation function.

In our scenario, the states $S$ correspond to the three question categories:
Simple, Complex, and Unsolvable. Actions are denoted as either reporting the
SLM answer or routing to the LLM. Observations, in the form of verifier output
$v$, enable the POMDP to ascertain its belief state, which is a probability
distribution over $S$. For instance, a high verifier confidence in the
correctness of $\mathcal{A}_{s}$ would increase the belief in the Simple
state. The solution to the POMDP subsequently yields a policy that maps belief
states to actions, effectively deciding whether to invoke the LLM based on a
balance of expected future rewards and computational costs. See Section B.1
for more details.

Another advantage of the POMDP-based meta-verifier is its interpretability and
customizability via reward assignment. For instance, in a Complex state,
assigning a very high reward of +50 for invoking the LLM indicates a
preference for accurate solutions over computational cost. Although the POMDP
framework inherently handles sequences of decisions, we confine our approach
to a single-decision scenario (horizon or episode length 1) for simplicity,
with potential for extension to streaming settings for optimizing across
multiple queries or a fixed time duration.

##  3 Cost-Performance Efficiency Analysis

In our approach to leveraging model performance, it is essential to consider
not only the raw accuracy of predictions but also the associated computational
or monetary costs. To that end, we introduce a metric to understand the
efficiency of the models in terms of cost. We use $C_{M}$ and $P_{M}$ to
denote the cost and performance of a method $M$. We also use
$C_{\textsc{SLM}}$ and $C_{\textsc{LLM}}$, and $P_{\textsc{SLM}}$ and
$P_{\textsc{LLM}}$, to denote the cost and performance of using the SLM and
LLM, respectively.

procedure AnswerQuery($\mathcal{C},q$)

$\triangleright$ $\mathcal{C}$: Context, $q$: Question,
$\textsc{SLM}/\textsc{LLM}$: Small/large language model

$\mathcal{A}_{s}\leftarrow\textsc{solve}(\textsc{SLM},\mathcal{C},q)$

$v\leftarrow\textsc{self-verify}(\mathcal{A}_{s},\mathcal{C},q)$

if $\textsc{meta-verify}(v,\mathcal{A}_{s},\mathcal{C},q)$ then

return $\mathcal{A}_{s}$

else

$\mathcal{A}_{l}\leftarrow\textsc{solve}(\textsc{LLM},\mathcal{C},q)$

return $\mathcal{A}_{l}$

end if

end procedure

$0$$5$$10$$15$$20$$25$$30$$35$$40$$45$$50$$55$$60$$50$$52$$54$$56$$58$$60$$62$$64$$66$$68$$70$CostPerformanceSLMLLM

Figure 3: Left: AutoMix algorithm. Right: Performance vs. Cost curve. The
slope between SLM and LLM provides a way to the Incremental Benefit per Cost
(ibc) for methods that mix models. Methods with a steeper slope than this
reference when plotted against SLM have a positive ibc (green region), whereas
those below the reference have a negative ibc (red region), falling into the
red region.

#### Incremental Benefit Per Cost (ibc)

We introduce methods, denoted by $M$, to optimally integrate SLM and LLM. For
each method $M$, we associate a cost $C_{M}$ and performance $P_{M}$. To
quantify the utility of $M$ over SLM, we define the metric Incremental Benefit
Per Cost (ibc) as $\textsc{ibc}_{M}$ (Equation 1).

$\displaystyle\textsc{ibc}_{M}$
$\displaystyle=\frac{P_{M}-P_{\textsc{SLM}}}{C_{M}-C_{\textsc{SLM}}},$
$\displaystyle\textsc{ibc}_{\textsc{base}}$
$\displaystyle=\frac{P_{\textsc{LLM}}-P_{\textsc{SLM}}}{C_{\textsc{LLM}}-C_{\textsc{SLM}}},$
$\displaystyle\Delta_{\textsc{ibc}}(M)$
$\displaystyle=\frac{\textsc{ibc}_{M}-\textsc{ibc}_{\textsc{base}}}{\textsc{ibc}_{\textsc{base}}}\times
100$ (1)

The ibc metric captures the efficiency of performance enhancement relative to
the additional cost. For comparative evaluation, we set a baseline ibc,
$\textsc{ibc}_{\textsc{base}}$, representing the benefit of always using LLM
over SLM. Finally, we compare methods using $\Delta_{\textsc{ibc}}$, which
compares the ibc of a specific method with $\textsc{ibc}_{\textsc{base}}$. A
positive ibc lift suggests that $M$ achieves performance increments more cost-
effectively than a standalone LLM, whereas a negative lift indicates reduced
efficiency (Figure 3)

#### Geometric Interpretation

On a Performance vs. Cost plot, consider the line segment joining the data
points of small language model (SLM) and large language model (LLM). This
segment’s slope represents a basic rate of performance increase for each
additional unit of cost. The Incremental Benefit per Cost (ibc) for any method
$M$ is the slope of the line from the SLM point to the point representing
$M$(Figure 3). A method $M$ that lies above the SLM-LLM segment provides a
steeper slope, indicating a favorable ibc (and a positive
$\Delta_{\textsc{ibc}}$). Conversely, if $M$ lies below the segment, it
suggests an unfavorable or negative ibc. Our primary objective is to identify
or develop methods that yield a consistently positive ibc, maximizing
performance enhancements for each additional unit of cost.

#### Cost Calculation

To evaluate the efficiency of a method $M$ that leverages both the Small
Language Model (SLM) and the Large Language Model (LLM), we define a cost
metric, $C_{M}$. This metric incorporates the costs of both initial answer
generation and verification by the SLM, as well as potential routing to the
LLM. Specifically, the total cost $C_{M}$ is computed as $C_{M}=2\times
C_{\text{SLM}}+w_{\text{LLM}}\times C_{\text{LLM}}$. Here, $C_{\text{SLM}}$
and $C_{\text{LLM}}$ represent the costs of a single query to the SLM and LLM,
respectively. The factor $w_{\text{LLM}}\in[0,1]$ denotes the proportion of
times the LLM is used, with $w_{\text{LLM}}=1$ indicating exclusive use and
$w_{\text{LLM}}=0$ denoting no usage. It’s important to note that while our
framework uses the SLM for verification, alternative verifiers could be
incorporated, which would adjust the cost formula accordingly.

While various complexities determine the pricing of these APIs (Dehghani et
al., 2021), given our emphasis on black-box utilization of large language
models, we choose to represent cost simply: the monetary expense charged to
the end user by the language model APIs.

##  4 Experiments

| CNLI | Quality | QASPER | NarrativeQA | COQA
---|---|---|---|---|---
Method | C | P | $\Delta_{\textsc{ibc}}$ | C | P | $\Delta_{\textsc{ibc}}$ | C | P | $\Delta_{\textsc{ibc}}$ | C | P | $\Delta_{\textsc{ibc}}$ | C | P | $\Delta_{\textsc{ibc}}$
SLM | 1 | 40.1 | - | 1 | 47.5 | - | 1 | 14.0 | - | 1 | 20.3 | - | 1 | 48.1 | -
FrugalGPT | 37.4 | 59.2 | 66.1 | 49.7 | 66.5 | -2.5 | 49.3 | 27.7 | -1.1 | 45.9 | 26.0 | 2.5 | 30.3 | 57.1 | 13.1
AutoMix w/ SC | $47.5$ | $52.3$ | $-17.0$ | $15.2$ | $52.8$ | $-7.0$ | $44.3$ | $26.8$ | $2.3$ | $23.0$ | $23.3$ | $9.2$ | $16.6$ | $54.7$ | 55.5
AutoMix w/ T | $51.9$ | $55.6$ | $-3.5$ | $37.7$ | $61.6$ | $-4.4$ | $47.2$ | $27.7$ | $3.7$ | $16.6$ | $22.4$ | 12.2 | $7.2$ | $50.7$ | $43.2$
AutoMix w/ P | $6.7$ | $43.5$ | 88.7 | $15.8$ | $52.9$ | $-11.8$ | $45.2$ | $27.6$ | 6.9 | $9.9$ | $21.4$ | $6.4$ | $6.9$ | $50.5$ | $43.7$
LLM | 50 | 55.5 | - | 50 | 67.1 | - | 50 | 28.1 | - | 50 | 26.4 | - | 50 | 61.4 | -

Table 1: Main Results: highlighting the trade-offs between Cost (C),
Performance (P), and Incremental Benefit per Cost ($\Delta_{\textsc{ibc}}$)
across various methods and datasets. The acronyms represent: SLM - Small
Language Model, LLM\- Large Language Model, AutoMix \+ T and AutoMix \+ P -
variations of our proposed method with thresholding (T) and POMDP (P) based
meta-verifiers, respectively. AutoMix \+ POMDP demonstrates a robust and
consistent $\Delta_{\textsc{ibc}}$ across cnli qasper, narrative-qa, and coqa
datasets, implying a judicious utilization of computational resources. AutoMix
outperforms FrugalGPT across all datasets, despite latter having access to
domain specific training and a near 0-cost verifier. While on cnli AutoMix \+
POMDP provides a lift of around 90%, on quality no variant of AutoMix or
baseline works. We discuss more of it in Analysis Section 4.2

#### Setup

We experiment with open-source pair llama2-13b and llama2-70b (Touvron et al.,
2023). We assume a cost of 1 unit for the SLM, and 50 units for the LLM,
following the price disparity between the small and large models offered by
LLM API providers like OpenAI and Together 222https://openai.com/pricing,
https://together.ai/. We refer readers to Appendix C for more details on
experimental setup.

#### Datasets

We experiment with several datasets, each with its unique context and
evaluation metric: i) narrative-qa (Kočiskỳ et al., 2018), which involves
question answering about full-length books and movie scripts (F1 score); ii)
qasper (Dasigi et al., 2021), focusing on question answering over research
papers (F1 score); iii) cnli (Koreeda and Manning, 2021), which targets
natural language inference tasks using non-disclosure agreements as context
and evaluates using accuracy; iv) quality (Pang et al., 2022), comprised of
multiple-choice questions from long articles and stories, evaluated on exact
match; and v) coqa (Reddy et al., 2019), consisting of conversational
comprehension questions that test models on coreference and pragmatic
reasoning (F1 score).

For evaluation, we utilize the validation sets from Shaham et al. (2022) for
narrative-qa, qasper, cnli, and quality, and use the prompts from Shaham et
al. (2023). For coqa, we employ its validation split and adapt the quality
prompt. Regardless of dataset, identical input prompts are dispatched to both
SLM and potentially LLM, ensuring consistent input processing costs. The
output length is fixed in multichoice datasets like cnli and quality, and the
brevity of responses in other datasets allows us to assume uniform output
processing costs. We use greedy decoding (temperature 0) and draw a single
sample for both the SLM and LLM.

#### Baselines

We use Frugal GPT (F) (Chen et al., 2023) as the baseline. We finetune a
DistillBert (Sanh et al., 2019) as a verifier, outputting a confidence
probability for a given question, context, and SLM-generated answer, with a
verifier confidence threshold directing query routing and its cost set to 0
due to significantly lower operational costs than SLM. Both approaches adhere
to a low-resource setting, utilizing 1000 training examples per dataset.

#### Proposed approaches

We experiment with three different types of meta-verifiers: i.) AutoMix \+
Self-Consistency: This method choses the majority decision from verifier from
32 drawn samples and performs the decision without any explicit meta-verifier.
ii) AutoMix + Thresholding: Using a threshold on the verifier probability
e.g., Thresh=0.75 implies using SLM outputs with confidence $\geq 0.75$ and
LLM. We use a threshold for each dataset that yields the highest
$\Delta_{\textsc{ibc}}$ on the validation set. iii) AutoMix + POMDP: This
method optimizes routing decisions using a POMDP solver Smith and Simmons
(2006) as a meta-verifier. The POMDP is learned on the validation set, and
takes decision based on the verifier outputs (detailed in Appendix B.1).

###  4.1 Main Results

Table 1 shows the meta-verifier method consistently showcases superior
performance in terms of $\Delta_{\textsc{ibc}}$ across both LLaMa2-13/70b. On
all datasets, AutoMix beat FrugalGPT despite the latter having access to
domain-specific training and low verifier cost. Further, on 3 of the 5
datasets, AutoMix-POMDP is the best performing method, with positive
improvement on all but qasper. We see maximum gains in coqa and cnli, with
AutoMix showing maximum improvement of 56% and 89% respectively. In Figure 4
(left), we present the performance of our model, AutoMix, across various cost
intervals. Our findings reveal that AutoMix-POMDP shows consistent positive
$\Delta_{\textsc{ibc}}$ across all evaluated costs. This suggests that our
method can deliver consistent improvements, regardless of the user’s desired
cost or performance requirements. Further, in Figure 4 (right), we compare the
accuracy of using POMDP based meta-verifier over Verifier-SC. We see
significant improvements across all datasets, with relative gains of up to 42%
demonstrating our proposed meta-verifier’s importance in few-shot verification
setups.

CNLIQualityQasperNarrativeCOQA$40$$50$$60$$70$$41$$70$$46$$64$$67$$58$$65$$54$$67$$71$Accuracy
(%)Verifier-SCPOMDP

Figure 4: Left: Aggregated performance vs. cost for different methods on the
small and large LLaMa2-13/70b. POMDP based meta-verifier is consistenly in the
green region, signifying a higher Incremental Benefit per Cost (ibc). Right:
The accuracy of the meta-verifier for both POMDP and Verifier-Self-Consistency
(Verifier-SC) approaches across various datasets. Across all scenarios, the
POMDP method consistently wins with up to 42% relative performance gains.

###  4.2 When and Why does meta-verification help?

Figure 5: Delta imporvements in F1-score of LLM over SLM for different values
of verifier probability. A perfect verifier should be a line with negative
slope: high delta when verifier confidence is low, and low delta when verifier
confidence is high. narrative-qa and coqa exhbit near perfect behavior. The
trend is reversed for cnli, with high confidence implying high delta. Unlike
others, quality show no correlation between train and test splits, explaining
poor ibc lifts, in learning based methods.

Figure 5 shows the relationship between the F1 score improvement b/w LLM and
SLM, denoted as $\Delta P_{LLM-SLM}$ (y-axis), for different verifier
confidence values (x-axis). Ideally, points with a high $\Delta P_{LLM-SLM}$
should be directed to LLM, as they result in significant gains in F1 score.
Consequently, a well-calibrated verifier is expected to exhibit a decreasing
linear trend: assign higher confidence to points where the gains from using a
LLM are lower. However, this expected behavior is only observed in the
narrative-qa and coqa datasets. In such scenarios, the necessity for a robust
meta-verifier is reduced as raw outputs from the verifier can be trusted. As a
result, self-verification performs well out-of-the-box with simple techniques
such as self-consistency and thresholding.

The verifier exhibits a peculiar behavior on the cnli dataset: the verifier’s
high confidence indicates a stronger performance of LLM over SLM. That is, the
verifier is more likely to suggest routing queries that will not gain much
from the LLM. In contrast, AutoMix with POMDP, informed by the validation set,
identifies this and adapts by discerning the optimal verifier probability
range for routing. This underscores the utility of the meta-verifier in
addressing verifier shortcomings.

On further investigation, we find that despite using identical prompts
(sourced from Shaham et al. (2023)), the llama2-13b model never answers
‘Entailment’, whereas llama2-70b never answers with ‘Contradiction’. While our
meta-verifier doesn’t directly process the llama2-generated answers, it learns
from the validation set that higher verifier confidence often corresponds to
the true answer being ‘Entailment’, leading to a preference for LLM routing.

#### When does AutoMix not work?

Analyzing the relatively poor performance of all methods on quality, we find a
substantial distribution shift between the training and testing splits for the
quality dataset in Figure 5. Consequently, AutoMix +POMDP overfits a policy on
the training set, which fails to generalize to the test set, resulting in a
performance that is inferior to AutoMix +SC. Further, neither variants of our
model nor the baselines exhibit a positive $\Delta_{\textsc{ibc}}$ for the
quality dataset. This is attributed to the lack of correlation between $\Delta
P_{LLM-SLM}$ and the verifier probability (Pearson coefficient = -0.03),
implying that the verifier provides no valuable signal. In this context, the
self-verifier’s performance is almost equivalent to a random guess, and the
meta-verifier also fails to help.

###  4.3 Key findings and takeaway

#### AutoMix is Effective in Low-Resource Scenarios

Figure 7 demonstrates the performance dynamics of AutoMix and FrugalGPT with
varying validation sizes. Notably, our method significantly outperforms
FrugalGPT with limited data (under 2000 samples), despite the latter’s domain-
specific training and zero verifier cost. However, as training data increases,
FrugalGPT narrows the performance gap by leveraging domain-specific training.
This pattern indicates that AutoMix provides a particularly advantageous
solution in real-world scenarios where data may be scarce.

#### Effectiveness of Few-shot Self-Verification

In Section A.1, we evaluate few-shot self-verification quantitatively and
qualitatively. We observe that the self-verification can effectively use
context to identify errors in answers generated by SLM in many cases.

#### Improving Self-Verification with Task-Specific Prompt Engineering

We explore the impact of task-specific prompt engineering on self-verification
performance in Section A.2. While prompt engineering improves verifier
accuracy, our meta-verifier remains robust in various settings and can
beneficially leverage even a weak verifier.

##  5 Related Work

#### Self-Verification

AutoMix aligns in spirit with works that aim to perform self-verification for
reasoning problems, such as Weng et al. (2023); Jiang et al. (2023) (see Pan
et al. (2023) for a survey of recent self-verification and correction
approaches). However, AutoMix uniquely harnesses context for verification
instead of relying on LLM’s knowledge (Dhuliawala et al., 2023) which can be
challenging for reasoning problems (Madaan et al., 2023; Huang et al., 2023),
and introduces a meta-verifier mechanism to offset the verifier’s potential
noise. Further, unlike Madaan et al. (2022), who utilize a corpus of past
mistakes to gauge the likelihood of a model error for a new question, AutoMix
uniquely utlizes context for verification. Our meta-verification approach can
also be seen in the context of conformal prediction (Angelopoulos et al.,
2023; Vovk et al., 2005) for a more robust self-verification. Ren et al.
(2023) demonstrated the efficacy of meta-verification techniques based on
conformal prediction in robot navigation, showing that layering predictions
from a language model with a secondary mechanism helps in identifying
situations that do not have adequate information for action.

#### Mixing Models

Distinct from related work optimizing LLM inference cost by model switching
and external verifiers Chen et al. (2023); Zhu et al. (2023); vSakota et al.
(2023), AutoMix obviates the need for verifier training through few-shot SLM
model prompting and does not require upfront access to all input queries. When
needed, the meta-verifier learned with only as few as 200 samples outperforms
training specialized models. Our work is thus aligned with recent work that
aims at composing different models and external tools for inference time
improvement of language models (Khattab et al., 2023; Press et al., 2022; Yao
et al., 2022; Zhou et al., 2022).

#### Adaptive Computation

In contrast to adaptive computation and model routing methods that preempt
computation via intermediate representations (Liu et al., 2020; Zhou et al.,
2020; Schuster et al., 2021; Geng et al., 2021; Schuster et al., 2022; Madaan
and Yang, 2022), AutoMix necessitates no architectural modifications and
assumes only black-box access to APIs. Further, unlike AdaptiveConsistency
(Aggarwal et al., 2023), which optimizes inference within a single LLM model,
AutoMix flexibly optimizes between two models and transcends its utility in
Self-Consistency.

##  6 Conclusion

AutoMix integrates black-box large language model (LLM) APIs into a multi-step
problem-solving framework, optimizing the computational cost and performance
trade-offs. AutoMix opens avenues for several interesting research directions.
First, while self-verification and correction are challenging for LLMs in
general, we find promising results using context-grounded few-shot
verification, indicating that similar approaches may yield gain in other
scenarios. Secondly, our work interweaves Good Old-Fashioned Artificial
Intelligence (GOFAI) approaches with LLMs, demonstrating that the
incorporation of a POMDP can boost the accuracy of a noisy few-shot verifier,
showing the promise of this paradigm as an approach for improving LLMs during
inference.

## References

  * Aggarwal et al. (2023) Pranjal Aggarwal, Aman Madaan, Yiming Yang, and Mausam. 2023.  Let’s sample step by step: Adaptive-consistency for efficient reasoning with llms.  _ArXiv_ , abs/2305.11860.
  * Angelopoulos et al. (2023) Anastasios N Angelopoulos, Stephen Bates, et al. 2023.  Conformal prediction: A gentle introduction.  _Foundations and Trends® in Machine Learning_ , 16(4):494–591.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901, Online. Curran Associates, Inc.
  * Chen et al. (2023) Lingjiao Chen, Matei A. Zaharia, and James Y. Zou. 2023.  Frugalgpt: How to use large language models while reducing cost and improving performance.  _ArXiv_ , abs/2305.05176.
  * Dagan et al. (2022) Ido Dagan, Dan Roth, Fabio Zanzotto, and Mark Sammons. 2022.  _Recognizing textual entailment: Models and applications_.  Springer Nature.
  * Dasigi et al. (2021) Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. 2021.  A dataset of information-seeking questions and answers anchored in research papers.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4599–4610.
  * Dehghani et al. (2021) Mostafa Dehghani, Yi Tay, Anurag Arnab, Lucas Beyer, and Ashish Vaswani. 2021.  The efficiency misnomer.  In _International Conference on Learning Representations_.
  * Dhuliawala et al. (2023) Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. 2023.  Chain-of-verification reduces hallucination in large language models.  _arXiv preprint arXiv:2309.11495_.
  * Dziri et al. (2023) Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jian, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D Hwang, et al. 2023.  Faith and fate: Limits of transformers on compositionality.  _arXiv preprint arXiv:2305.18654_.
  * Geng et al. (2021) Shijie Geng, Peng Gao, Zuohui Fu, and Yongfeng Zhang. 2021.  Romebert: Robust training of multi-exit bert.  _arXiv preprint arXiv:2101.09755_.
  * Huang et al. (2023) Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2023.  Large language models cannot self-correct reasoning yet.  _arXiv preprint arXiv:2310.01798_.
  * Jiang et al. (2023) Weisen Jiang, Han Shi, Longhui Yu, Zhengying Liu, Yu Zhang, Zhenguo Li, and James T Kwok. 2023.  Backward reasoning in large language models for verification.  _arXiv preprint arXiv:2308.07758_.
  * Khattab et al. (2023) Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, et al. 2023.  Dspy: Compiling declarative language model calls into self-improving pipelines.  _arXiv preprint arXiv:2310.03714_.
  * Kočiskỳ et al. (2018) Tomáš Kočiskỳ, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018.  The narrativeqa reading comprehension challenge.  _Transactions of the Association for Computational Linguistics_ , 6:317–328.
  * Koreeda and Manning (2021) Yuta Koreeda and Christopher D Manning. 2021.  Contractnli: A dataset for document-level natural language inference for contracts.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 1907–1919.
  * Kwon et al. (2023) Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023.  Efficient memory management for large language model serving with pagedattention.  In _Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles_.
  * Le Scao and Rush (2021) Teven Le Scao and Alexander M Rush. 2021.  How Many Data Points is a Prompt Worth? In _NAACL_.
  * Liu et al. (2021a) Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021a.  What Makes Good In-Context Examples for GPT-3? _arXiv:2101.06804 [cs]_.  ArXiv: 2101.06804.
  * Liu et al. (2021b) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021b.  Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing.  _arXiv preprint arXiv:2107.13586_.
  * Liu et al. (2020) Weijie Liu, Peng Zhou, Zhiruo Wang, Zhe Zhao, Haotang Deng, and Qi Ju. 2020.  Fastbert: a self-distilling bert with adaptive inference time.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6035–6044.
  * Madaan et al. (2022) Aman Madaan, Niket Tandon, Peter Clark, and Yiming Yang. 2022.  Memory-assisted prompt editing to improve GPT-3 after deployment.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 2833–2861, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Madaan et al. (2023) Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2023.  Self-refine: Iterative refinement with self-feedback.  _arXiv preprint arXiv:2303.17651_.
  * Madaan and Yang (2022) Aman Madaan and Yiming Yang. 2022.  Flowgen: Fast and slow graph generation.  _arXiv preprint arXiv:2207.07656_.
  * Mishra et al. (2021) Swaroop Mishra, Daniel Khashabi, Chitta Baral, Yejin Choi, and Hannaneh Hajishirzi. 2021.  Reframing Instructional Prompts to GPTk’s Language.  _arXiv preprint arXiv:2109.07830_.
  * Monahan (1982) George E. Monahan. 1982.  State of the art—a survey of partially observable markov decision processes: Theory, models, and algorithms.  _Management Science_ , 28:1–16.
  * Pan et al. (2023) Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. 2023.  Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies.  _arXiv preprint arXiv:2308.03188_.
  * Pang et al. (2022) Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, et al. 2022.  Quality: Question answering with long input texts, yes!  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5336–5358.
  * Poliak (2020) Adam Poliak. 2020.  A survey on recognizing textual entailment as an nlp evaluation.  _arXiv preprint arXiv:2010.03061_.
  * Press et al. (2022) Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. 2022.  Measuring and narrowing the compositionality gap in language models.  _arXiv preprint arXiv:2210.03350_.
  * Reddy et al. (2019) Siva Reddy, Danqi Chen, and Christopher D Manning. 2019.  Coqa: A conversational question answering challenge.  _Transactions of the Association for Computational Linguistics_ , 7:249–266.
  * Reid and Neubig (2022) Machel Reid and Graham Neubig. 2022.  Learning to model editing processes.  _arXiv preprint arXiv:2205.12374_.
  * Ren et al. (2023) Allen Z Ren, Anushri Dixit, Alexandra Bodrova, Sumeet Singh, Stephen Tu, Noah Brown, Peng Xu, Leila Takayama, Fei Xia, Jake Varley, et al. 2023.  Robots that ask for help: Uncertainty alignment for large language model planners.  _arXiv preprint arXiv:2307.01928_.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  _arXiv preprint arXiv:1910.01108_.
  * Schick et al. (2022) Timo Schick, Jane Dwivedi-Yu, Zhengbao Jiang, Fabio Petroni, Patrick Lewis, Gautier Izacard, Qingfei You, Christoforos Nalmpantis, Edouard Grave, and Sebastian Riedel. 2022.  Peer: A collaborative language model.
  * Schuster et al. (2022) Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Q Tran, Yi Tay, and Donald Metzler. 2022.  Confident adaptive language modeling.  _arXiv preprint arXiv:2207.07061_.
  * Schuster et al. (2021) Tal Schuster, Adam Fisch, Tommi Jaakkola, and Regina Barzilay. 2021.  Consistent accelerated inference via confident adaptive transformers.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 4962–4979.
  * Shaham et al. (2023) Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. 2023.  Zeroscrolls: A zero-shot benchmark for long text understanding.
  * Shaham et al. (2022) Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, and Omer Levy. 2022.  SCROLLS: Standardized CompaRison over long language sequences.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 12007–12021, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Shinn et al. (2023) Noah Shinn, Beck Labash, and Ashwin Gopinath. 2023.  Reflexion: an autonomous agent with dynamic memory and self-reflection.
  * Smith and Simmons (2006) Trey Smith and Reid Simmons. 2006.  Focused real-time dynamic programming for mdps: Squeezing more out of a heuristic.  In _AAAI_ , pages 1227–1232.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Vovk et al. (2005) Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. 2005.  _Algorithmic learning in a random world_ , volume 29.  Springer.
  * vSakota et al. (2023) Marija vSakota, Maxime Peyrard, and Robert West. 2023.  Fly-swat or cannon? cost-effective language model choice via meta-modeling.  _ArXiv_ , abs/2308.06077.
  * Welleck et al. (2022) Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. 2022.  Generating sequences by learning to self-correct.  _arXiv preprint arXiv:2211.00053_.
  * Weng et al. (2023) Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Kang Liu, and Jun Zhao. 2023.  Large language models are better reasoners with self-verification.  _CoRR, abs/2212.09561_.
  * Yao et al. (2022) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2022.  React: Synergizing reasoning and acting in language models.  In _The Eleventh International Conference on Learning Representations_.
  * Zhou et al. (2022) Shuyan Zhou, Uri Alon, Frank F Xu, Zhengbao Jiang, and Graham Neubig. 2022.  Docprompting: Generating code by retrieving the docs.  In _The Eleventh International Conference on Learning Representations_.
  * Zhou et al. (2020) Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei. 2020.  Bert loses patience: Fast and robust inference with early exit.  _Advances in Neural Information Processing Systems_ , 33:18330–18341.
  * Zhu et al. (2023) Banghua Zhu, Ying Sheng, Lianmin Zheng, Clark W. Barrett, Michael I. Jordan, and Jiantao Jiao. 2023.  On optimal caching and model multiplexing for large model inference.  _ArXiv_ , abs/2306.02003.

##  Appendix A Verifier Qualitative Analysis

###  A.1 How effective is few-shot self-verification?

One notable contribution of this work is the concept of few-shot self-
verification of outputs. Self-Verification, especially for reasoning problems,
poses its own set of challenges; however, our setup has a unique advantage:
the capacity to utilize context to validate answers. For instance, the model
can identify factual inaccuracies in the answer or discern apparent
contradictions that might not have been evident during the initial response.
But does this advantage translate to effective self-verification in practice?
As depicted in Figure 6, aside from the CNLI dataset, few-shot self-
verification succeeds in accurately identifying correct examples by assigning
them higher probabilities across all other datasets.

#### Qualitative Analysis

Representative Examples from our qualitative analysis are shown in Tables 2,
3, and 4.

Attribute |  Content
---|---
Context |  Which country grows the most tea? … Shen sipped it. He liked it. He drank it all. Shen was proud of his new drink. Truncated for brevity
Question |  When did he take it?
Generated Answer |  He took it in 1990.
Gold Answer |  about 2737 B.C
Verifier Output |  The context does not mention that he took tea in 1990. Verification Decision: The AI generated answer is Incorrect.
Table 2: An example where the generated answer is verifiably incorrect, citing an unsupported date (1990), and the verifier successfully catches this discrepancy. Attribute |  Content
---|---
Context |  Fort Lauderdale, Florida (CNN) – Just taking a sip of water or walking to the bathroom is excruciatingly painful for 15-year-old Michael Brewer, … He suffered second- and third-degree burns over about two-thirds of his body, according to the hospital’s associate director, Dr. Carl Schulman. Truncated for brevity
Question |  How long before he recovers from his injury?
Generated Answer |  he will never recover from his injury.
Gold Answer |  He has a lifelong recovery.
Verifier Output |  The context clearly states that the teen is recovering from his injuries. Verification Decision: The AI generated answer is Incorrect.
Table 3:  The example demonstrates a challenging case for the verifier. The generated answer is a rephrasing of what’s mentioned in the context, however the verifier fails to catch it. Attribute |  Content
---|---
Context |  Pruning keeps the plant only three or four feet tall. This is an easy height for tea picking. Green tea is steamed right after the leaves are picked. Green tea _ its healthy qualities. For example, it may prevent heart disease. How did we get tea bag? The answer: by accident. Truncated
Question |  What did they do to green tea after picking it?
SLM Answer |  steamed.
Gold Answer |  prune it
Verifier Output |  The context mentions that green tea is steamed after being picked. Verification Decision: The AI generated answer is Correct.
Table 4: An instance where the verifier deems the answer correct, and the gold
label was incorrect. The verifier is able to correctly infer that the context
mentions tea being steamed after picking. Figure 6: Verifier Probability and
Correctness: Percentage of correct responses across distinct verifier
probability bins, representing
$P(\mathcal{C}=1\,|\,A_{\textsc{SLM}},\mathcal{C},q)$, where
$A_{\textsc{SLM}}$ is the answer from the Small Language Model, $\mathcal{C}$
is the context, and $q$ is the query. Each bin represents a range of verifier
probabilities and the corresponding accuracy of the responses within that
probability range across various datasets. Notably, for all datasets,
excluding cnli and qasper, a higher verification score generally corresponds
to a larger proportion of correct examples, indicating that the verifier is,
to an extent, capable of discerning the reliability of responses generated by
itself. We use a meta-verifier to get around these noisy predictions.

#### FrugalGPT vs. AutoMix at different levels of data availability

Figure 7 demonstrates the performance dynamics of AutoMix and FrugalGPT with
varying validation data sizes. Notably, our method significantly outperforms
FrugalGPT with limited data (under 2000 samples), despite the latter’s domain-
specific training and zero verifier cost. However, as training data increases,
FrugalGPT narrows the performance gap by leveraging its domain-specific
training. This pattern indicates that AutoMix provides a particularly
advantageous solution in real-world scenarios where data may be scarce.

Figure 7: Comparison of AutoMix with FrugalGPT over varying Training Dataset
Size. Despite zero-cost verifier and domain-specific training, FrugalGPT
underperforms AutoMix. AutoMix is especially useful for limited data settings,
with higher gains visible when dataset size is less than 1000.

###  A.2 Domain-specific vs. Domain independent verifier

We used a single verifier with the llama2-13b model to help steer the model.
To avoid excessive prompt engineering, we used a generic prompt for all
datasets. However, task-specific prompts generally help (Le Scao and Rush,
2021; Liu et al., 2021b; Mishra et al., 2021; Liu et al., 2021a). To
investigate this, we create task specific prompts for cnli by giving examples
from legal domain in the prompt.

Figure 8 underscores the efficacy of employing task-specific verification
prompts, ensuring a heightened probability allocation for accurate examples
during the verification process. Interestingly, the enhanced verifier accuracy
does not always directly translate to proportionate improvements in our
proposed method, AutoMix, as evidenced in Table 5. This phenomenon higlights
the role of meta-verifiers, adeptly negotiating through the outputs of
potentially unreliable verifiers.

Figure 8: Enhancement of verifier accuracy using task-specific verification
prompts, which allocate higher verification probabilities to more correct
examples.

| CNLI | CNLI-CV
---|---|---
Method | Cost | Perf. | IBC_Lift | Cost | Perf. | IBC_Lift
SLM | 1 | 40.1 | - | 1 | 40.1 | -
FrugalGPT | 37.4 | 59.2 | 66.1 | 37.4 | 59.2 | 66.1
Self-Consistency | 47.5 | 52.3 | -17.0 | 40.5 | 50.6 | -15.5
AutoMix-Threshold | 51.9 | 55.6 | -3.5 | 28.1 | 46.9 | -49.1
AutoMix-POMDP | 6.7 | 43.5 | 88.7 | 15.8 | 45.2 | 12.4
LLM | 50 | 55.5 | - | 50 | 55.5 | -

Table 5: Despite the boost in verifier accuracy with task-specific prompts
(Figure 8), AutoMix may not always benefit, highlighting the utility of even
weak verifiers when supported by meta-verifiers.

###  A.3 Meta-Verification Analaysis

Figure 9: Delta imporvements in F1-score of LLM over SLM for different values
of verifier probability for qasper. While there are small regions where there
is overall incrememental benefit, routing queries where SLM outputs
unanswerable or when verifier is confident is helpful.

In Section 4.2, we discussed importance of meta-verifier for different
datasets. Here we also discuss the case for qasper dataset. Apart from
standard QA, qasper also requires models to identify questions that are
unanswerable from the given context. However, when a SLM outputs
‘Unanswerable’, it is possible, that it is not able to figure out the answer
from context, instead of the question actually being unanswerable. Therefore,
we route all such queries to the LLM without consulting a verifier. Figure 9
shows the $\Delta P_{LLM-SLM}$vs verifier probability for qasper.
Interestingly, train and test show strong correlation for all except lower
confidence thresholds. Further, the figure shows routing unanswerable queries
directly to LLM is useful as it results in higher than average F1 gain.

##  Appendix B Methodology

###  B.1 POMDP

The Partially Observable Markov Decision Process (POMDP) presents a robust
framework for handling decision-making problems under uncertainty, offering a
structured way to manage and navigate through the decision spaces where the
system’s state is not fully observable Monahan (1982). A POMDP is defined by a
tuple $(S,A,T,R,\Omega,O)$, where $S$ is a set of states, $A$ is a set of
actions, $T$ represents the state transition probabilities, $R$ is the reward
function, $\Omega$ is a set of observations, and $O$ is the observation
function.

In the context of meta-verifier, the unobservable states ($S$) represent the
potential correctness of the verifier’s predictions, categorized as Simple,
Complex, and Insolvable. Actions ($A$) are binary: trust the verifier or
invoke the LLM. The reward function ($R$) quantifies the cost or gain of
making a particular action in a given state, steering the decision policy
towards cost-effective actions. Observations ($\Omega$) in our model are the
verifier’s probability outputs, discretized into bins. Specifically, we
generate k=8 samples from the verifier, discretizing our observation space in
intervals of size 0.125 ranging from 0 to 1.

The observation function ($O$) depicts the likelihood of observing an
observation given an action was taken and the system transitioned to a
particular state. Using an appropriate observation function is crucial for
POMDP to work. Specifically, we define observations probabilities in two ways:

  * •

1\. Functional Form: For each of the states $s$, the observation function $O$
is defined as $O(s,v)=\frac{1}{K}\cdot v^{\gamma_{s}}$, where $v$ is the
verifier probability and $\gamma_{s}\in[0,\infty]$ is a hyperparameter for
every state and $K$ is normalizing factor. Intutively, a value of $\gamma$
close to 1 indicates ideal calibration, with verifier probability $v$
indicating true probability of being in a particular state. The values of
$\gamma_{s}$’s for the three states are determined based on the respective
POMDP’s performance on validation set based on the IBC-Lift.

  * •

2\. Discrete Form: An alternate option is to directly learn observation
function $O$ from the statistics of validation set. Since in validation set,
we have access to the true state along with verifier probabilites of
individual data instances, we can model observation function as
$O(s,v)=\frac{\Sigma_{i=0}^{N}\mathbf{1}\\{s_{i}=s\;and\;v_{i}=v\\}}{\Sigma_{i=0}^{N}\mathbf{1}\\{s_{i}=s\\}}$.
The method has the advantage of being hyperparameter free and provides more
accurate representation by computing the true observation probabilities on
validation set. However, it performs worse than functional form, when either
certain values of $v$ or $s$ are not well represented in validation set or in
cases of high distribution shift between validation and test set.

Since both these methods have their strengths, and are independent of each
other, we choose the best performing method on validation set.

This POMDP mechanism allows for optimal decision-making under uncertainty,
balancing the cost and reliability of invoking the LLM. Through employing
standard POMDP solving algorithms such as Focused Real-Time Dynamic
Programming333We use zmdp package https://github.com/trey0/zmdp for solving
POMDP Smith and Simmons (2006), we derive a policy that maps belief states
(probability distributions over $S$) to actions. During inference, the learned
policy effectively decides whether to trust the verifier’s output or to invoke
the LLM based on a combination of expected future rewards and computational
costs.

Another advantage of the POMDP-based meta-verifier is its interpretability and
customizability via reward assignment. For instance, in a "Needy" state,
assigning a reward of +50 for invoking the LLM indicates a preference for
accurate solutions over computational cost. Conversely, in a "Good" state,
designating a reward of -10 for trusting the SLM encourages computational
savings. This enables users to strategically balance solution quality against
computational expenses, aligning with specific application needs.

⬇

# Meta-verifier POMDP File for narrative_qa

\pardiscount: 0.99

values: reward

\par# We have 6 states: 3 corresponding to the initial state before verifier
is called, and 3 corresponding to the state after verifier is called

states: START_S START_C START_U SIMPLE COMPLEX UNSOLVABLE

\par# Effectively, we have 3 actions: 1.) The initial State where we run
verifier 2.) Report SLM’s Answer 3.) Invoke LLM and Report its Answer

actions: Init Trust_SLM Invoke_LLM

\par# Observations lies in one of verifier probability bins. Eg:
bin_correct_high represents Verifier outputs SLM answer as correct with high
confidence

observations: bin_incorrect_low bin_incorrect_high bin_correct_low
bin_correct_high

\par# Transition Model for Init action

\parT: Init

# Format: start_state : end_state : Transition_Probability

\par# Transition Model for Trust_SLM action

T: Trust_SLM

identity

\par# Transition Model for Invoke_LLM action

T: Invoke_LLM

identity

\par# Observation Model after "Init" action for narrative_qa

# Format: O : action : state : observation : probability

\par# Eaxmple: In SIMPLE cases, it is likely, SLM is correct and Verifier is
Confident, while in UNSOLVABLE, SLM is incorrect (Lower Obs. Probability)

O : * : SIMPLE : bin_correct_high 0.8

O : * : COMPLEX : bin_correct_high 0.4

O : * : UNSOLVABLE : bin_correct_high 0.1

\par# Reward Model:

# Format: R: action : init_state : end_state : observation : probability

\par# Example: For COMPLEX state, Trusting SLM results in negative score,
while invoking LLM results in a high +50 score.

R: Trust_SLM : COMPLEX : * : * -10

R: Invoke_LLM : COMPLEX : * : * +50

Figure 10: A sample POMDP specification file. POMDP requires defining states,
actions, observations and relevant Transition, Observation Probabilities and
Reward Values.

##  Appendix C Additional Details on the Experimental Setup

For evaluation, we utilize the validation sets from Shaham et al. (2022) for
narrative-qa, qasper, cnli, and quality, along with their provided prompts.
For coqa, we employ its validation split and adapt the quality prompt. For
consistency, 1000 instances are sampled from val set of each dataset. The
procedure is repeated over 10 seeds, to reduce variance. Regardless of
dataset, identical input prompts are dispatched to both SLM and potentially
LLM, ensuring consistent input processing costs. The output length is fixed in
multichoice datasets like cnli and quality, and the brevity of responses in
other datasets allows us to assume uniform output processing costs. We use
greedy decoding (temperature 0) and draw a single sample for both the SLM and
LLM. For verification, we generate 8 samples per question, which has
negligible cost owing to large context. In analysis section 4.2, we draw 32
samples from verifier and apply gaussian smoothing to curves for convenient
visualization without varying noise.

For running our experiments we use llama2-13b and llama2-70b models from
huggingface444Models available at: https://huggingface.co/meta-
llama/Llama-2-13b-hf and https://huggingface.co/meta-llama/Llama-2-70b-hf. We
use vllm Kwon et al. (2023) for hosting models for inference.

##  Appendix D Few-Shot Prompts

⬇

Story:

{relevant parts of the story}

\par{instruction}

\parQuestion: {question}

\parAnswer:

List of codeblocks 2 Task Prompt. We experiment with long-context reasoning
tasks, which require answering questions from stories, legal contracts,
research papers, and novels.

⬇

Context: {context}

\parQuestion: {question}

\parAI Generated Answer: {generated_answer}

\parInstruction: Your task is to evaluate if the AI Generated Answer is
correct, based on the provided context and question. Provide the judgement and
reasoning for each case. Choose between Correct or Incorrect.

\parEvaluation:"’

List of codeblocks 3 Verification Prompt. The verification process is framed
as a natural language entailment task, where the model determines the validity
of the model-generated answer with respect to the context and question.

###  D.1 Verifier Prompts

⬇

Context: The manuscript, discovered in 1980 in a dusty attic, turned out to be
a lost work of Shakespeare.

\parQuestion: Whose lost work was discovered in a dusty attic in 1980?

\parAI Generated Answer: Shakespeare

\parInstruction: Your task is to evaluate if the AI Generated Answer is
correct, based on the provided context and question. Provide the judgement and
reasoning for each case. Choose between Correct or Incorrect.

\parEvaluation: The context specifically mentions that a lost work of
Shakespeare was discovered in 1980 in a dusty attic.

\parVerification Decision: The AI generated answer is Correct.

\par—

\parContext: The celestial event, known as the Pink Moon, is unique to the
month of April and has cultural significance in many indigenous tribes.

\parQuestion: In which month does the celestial event, the Pink Moon, occur?

\parAI Generated Answer: July

\parInstruction: Your task is to evaluate if the AI Generated Answer is
correct, based on the provided context and question. Provide the judgement and
reasoning for each case. Choose between Correct or Incorrect.

\parEvaluation: The context clearly states that the Pink Moon is unique to the
month of April.

\parVerification Decision: The AI generated answer is Incorrect.

\par—

\par{truncated examples}

\parContext: {context}

\parQuestion: {question}

\parAI Generated Answer: {generated_answer}

\parInstruction: Your task is to evaluate if the AI Generated Answer is
correct, based on the provided context and question. Provide the judgement and
reasoning for each case. Choose between Correct or Incorrect.

\parEvaluation:

List of codeblocks 4 Few-Shot Verifier Prompts: 3-shot verifier prompt for
evaluating the correctness of SLM’s answer. The same prompt is used for all
datasets.


# Cell Spatial Analysis in Crohn’s Disease: Unveiling Local Cell Arrangement
Pattern with Graph-based Signatures

Shunxing Bao  Department of Electrical and Computer Engineering, Vanderbilt
University, Nashville, TN, USA  Sichen Zhu  Department of Biomedical
Engineering, Georgia Institute of Technology and Emory University, Atlanta,
GA, USA  Vasantha L Kolachala  Division of Pediatric Gastroenterology,
Department of Pediatrics and Pediatric Research Institute, Children’s
Healthcare of Atlanta, Emory University School of Medicine, Atlanta, GA, USA
Lucas W. Remedios  Department of Computer Science, Vanderbilt University,
Nashville, TN, USA  Yeonjoo Hwang  Division of Pediatric Gastroenterology,
Department of Pediatrics and Pediatric Research Institute, Children’s
Healthcare of Atlanta, Emory University School of Medicine, Atlanta, GA, USA
Yutong Sun  School of Electrical and Computer Engineering, Georgia Institute
of Technology, Atlanta, GA, USA  Ruining Deng  Department of Computer Science,
Vanderbilt University, Nashville, TN, USA  Can Cui  Department of Computer
Science, Vanderbilt University, Nashville, TN, USA  Yike Li  Department of
Otolaryngology-Head and Neck Surgery, Vanderbilt University Medical Center,
Nashville, TN, USA  Jia Li  Department of Biostatistics, Vanderbilt University
Medical Center, Nashville, TN, USA  Joseph T. Roland  Department of Surgery,
Vanderbilt University, Medical Center, Nashville, TN, USA  Qi Liu  Department
of Biostatistics, Vanderbilt University Medical Center, Nashville, TN, USA
Ken S. Lau  Department of Cell and Developmental Biology, Vanderbilt
University School of Medicine, Nashville, TN, USA  Subra Kugathasan  Division
of Pediatric Gastroenterology, Department of Pediatrics and Pediatric Research
Institute, Children’s Healthcare of Atlanta, Emory University School of
Medicine, Atlanta, GA, USA  Peng Qiu  Department of Biomedical Engineering,
Georgia Institute of Technology and Emory University, Atlanta, GA, USA  Keith
T. Wilson  Division of Gastroenterology, Hepatology, and Nutrition, Department
of Medicine, Vanderbilt University Medical Center, Nashville, TN, USA
Veterans Affairs Tennessee Valley Healthcare System, Nashville, TN, USA  Lori
A. Coburn  Division of Gastroenterology, Hepatology, and Nutrition, Department
of Medicine, Vanderbilt University Medical Center, Nashville, TN, USA
Veterans Affairs Tennessee Valley Healthcare System, Nashville, TN, USA
Bennett A. Landman  Department of Electrical and Computer Engineering,
Vanderbilt University, Nashville, TN, USA  Department of Computer Science,
Vanderbilt University, Nashville, TN, USA  Yuankai Huo  Department of
Electrical and Computer Engineering, Vanderbilt University, Nashville, TN, USA
Department of Computer Science, Vanderbilt University, Nashville, TN, USA

###### Abstract

Crohn’s disease (CD) is a chronic and relapsing inflammatory condition that
affects segments of the gastrointestinal tract. CD activity is determined by
histological findings, particularly the density of neutrophils observed on
Hematoxylin and Eosin stains (H&E) imaging. However, understanding the broader
morphometry and local cell arrangement beyond cell counting and tissue
morphology remains challenging. To address this, we characterize six distinct
cell types from H&E images and develop a novel approach for the local spatial
signature of each cell. Specifically, we create a 10-cell neighborhood matrix,
representing neighboring cell arrangements for each individual cell. Utilizing
t-SNE for non-linear spatial projection in scatter-plot and Kernel Density
Estimation contour-plot formats, our study examines patterns of differences in
the cellular environment associated with the odds ratio of spatial patterns
between active CD and control groups. This analysis is based on data collected
at the two research institutes. The findings reveal heterogeneous nearest-
neighbor patterns, signifying distinct tendencies of cell clustering, with a
particular focus on the rectum region. These variations underscore the impact
of data heterogeneity on cell spatial arrangements in CD patients. Moreover,
the spatial distribution disparities between the two research sites highlight
the significance of collaborative efforts among healthcare organizations. All
research analysis pipeline tools are available at
https://github.com/MASILab/cellNN.

###### keywords:

Cell spatial analysis, Pattern recognition, Crohn’s disease.

##  1 INTRODUCTION

Crohn’s disease (CD) is a complex inflammatory bowel disease (IBD) affecting
the gastrointestinal tract, characterized by persistent and recurring bowel
inflammation [1]. The prevalence of IBD has been on the rise, leading to
increased medical expenditures. Notably, the medical cost of CD was estimated
to be $3.48 billion per year in 2015, and it is projected to reach $3.72
billion per year by 2025, constituting a significant portion of the overall US
national costs [2]. The Gut Cell Atlas Crohn’s Disease Consortium is an
ambitious initiative supported by The Leona M. and Harry B. Helmsley
Charitable Trust, aiming to develop comprehensive cellular reference maps for
CD. The primary focus of this initiative is to compare tissues from CD
patients in comparison to healthy controls
(https://www.gutcellatlas.helmsleytrust.org/). By mapping different human cell
types and analyzing gene and protein expression in the context of response to
anatomical locations and CD, this project offers a unique opportunity to
advance our understanding of the human gut and its implications in CD.

In tumor research, spatial analysis using H&E-stained images has been
extensively explored. For instance, Failmezger et al. revealed key tumor
microenvironment features through topological tumor graphs in melanoma
specimens [3], while Xu et al. assessed tumor mutational burden and immune
infiltrates in bladder cancer patients using H&E and iHC imaging [4]. They
also developed a method to detect prognostic tumor infiltrating lymphocytes
(TIL) density in colorectal carcinoma patients [5]. Saltz et al. generated TIL
maps from H&E images, correlating them with survival in diverse tumor types
[6]. However, applying spatial analysis to inflammatory bowel disease (IBD)
remains under investigation.

Determination of CD activity depends on histological findings, particularly
the density of neutrophils observed via hematoxylin and eosin (H&E) staining
[7]. There are many ways to assess the samples from CD patients. For instance,
pathologists can understand the cell neighborhood changes with a zoom in and
out on the biopsies. The invasion of neutrophils is known to be a sign of
active inflammation and is a well-known problem [8]. Pathologist-assigned
disease severity scores for CD biopsies are often given at the slide level,
though the disease features that resulted in the scoring might not present
homogeneously across the slide. As Figure 1 shows, we can see that the high
density of neutrophils is not shown in all areas of the tissue on a slide. Can
we quantify the relationship of between cell cells, beyond the magnitude of
the neutrophils or other cell types is the primary motivation of this work.

Figure 1: We present four patches collected from two sample slides, where one
slide is CD normal, and the other slide is diagnosed as CD active. Neutrophils
can be found in both types of the tissues. Counting the density of neutrophils
is one of the pivotal biomarkers used in identify CD activity. Additionally,
pathologists have the capability to zoom in and observe the morphology of
individual nuclei or clusters. Our objective is to explore and quantify a more
comprehensive morphometric pattern within the localized cellular arrangement.

In this study, we deviate from conventional morphological feature-based CD
activity classification. Instead, our focus is on delving into and defining a
graph-based metric aimed at enabling spatial analysis. Our core hypothesis is
that distinct patterns might characterize the relationships among various cell
types within the context of the CD. Utilizing established tools to identify
six specific cell types from H&E images, we introduce an innovative approach
to capture the local spatial characteristics of each cell. In brief, we create
a 10-cell neighborhood matrix that outlines the arrangement of neighboring
cells for each individual cell. Through the utilization of t-SNE for non-
linear spatial projection in both scatter-plot and Kernel Density Estimation
(KDE) contour-plot formats, our research delves into disparities in cellular
environments, particularly those linked to the odds ratio of spatial patterns
between CD active and control groups. The contribution of this study is three-
fold:

  * •

We propose a graph-based signature to represent a local arrangement signature
for each cell.

  * •

We develop a comprehensive visualization workflow to identify the relationship
of signature patterns among active CD and normal CD, and healthy control
cohorts.

  * •

We present investigations from two research institutes with heterogeneous data
acquisition, focusing on the rectum.

##  2 Method

Our objective is to comprehend alterations in the cellular neighborhood. To
accomplish this, it is essential to establish biomarkers capable of detecting
the nuanced local histological orientation and interrelationships among co-
located cells. Here, we investigate a potential avenue for achieving this
goal. Initially, we outline a graph-based spatial characterization for each
cell. Subsequently, we introduce a visualization technique designed to handle
a substantial cell count. Lastly, the quantification method is elucidated. The
comprehensive analysis workflow is depicted in Figure 3.

###  2.1 Data pre-processing

Prior to generating the spatial signature of each cell, we first segment the
whole slide image (WSI) using a pre-trained segmentation deep learning model
(HoverNet [9]) from the Colon Nuclei Identification and Counting (CoNIC)
Challenge dataset [10], which can identify six cell types: neutrophils,
epithelial cells, lymphocytes, plasma cells, eosinophils, and connective
tissue. The HoverNet CoNIC pre-trained segmentation model operates only on
patches with size of 256$\times$256 pixels under 20$\times$ magnification. So
we we employ the CLAM [11] method to remove the background of the WSI and
divide the gigapixel image into relevant size of patches, segment each patch
(in 20$\times$) using the pre-trained model, and then merge all the patches
back into the original WSI space to collect the global coordinate of the
cells.

###  2.2 Graph-based spatial signature

For building the spatial signature, we aim to find 11-nearest neighbor for
each cell, converting the 11 neighbors into a count matrix for the six cell
types. Figure 2 depicts two cell samples of the same type but with different
local arrangements. To achieve this, we utilize the KD-tree algorithm [12,
13], which creates a binary tree partitioning the cell coordinate space into
smaller regions for fast searching of nearest neighbor points in multi-
dimensional spaces. Each cell is assigned with 11 indexes, including itself,
resulting in a total count of 10 when excluding the cell itself. To clean up
feature noise, we remove edge cases of cells containing less than 10 exclusive
neighbors (10-NN).

Figure 2: Counting the density of neutrophils is one of the key biomarkers to
identify CD, which is a well-known problem. We would like to understand if
there is a broader morphometry existing within cell local arrangement.

###  2.3 Visualization

We aggregate the count matrix of 10-NN signatures for all cells in the target
dataset, comprising CD active and control groups, into a large matrix,
considering that each WSI may contain over ten thousand cells. Subsequently,
we utilize a 2-D standard t-SNE with the KL divergence as the cost function
[14]. The t-SNE technique effectively maintains the relative distances between
neighboring data points, accentuating local structure over global structure.
Initially, when employing t-SNE scatter plots, it becomes straightforward to
visualize the distinct exclusive regions for the two categories. However, due
to the vast number of data points, numerous overlaps are expected, making it
difficult to convey the data point density through the scatter plot. As a
result, we opt to transform the t-SNE embedding into KDE contour plots to
identify regions of interest with diverse probabilities. This visualization
approach establishes a non-linear spatial space encompassing all cell data
points from the input whole dataset.

###  2.4 Quantification

We investigate any regions of interest (ROI) of the t-SNE visualization using
bounding box (BBox). To comprehend the occurrence probability of the specific
10-nearest neighbor (10-NN) pattern in two different CD activity groups, we
calculate the odds ratio. The odds ratio is defined as the fraction of cells
from CD activity group 1 divided by the fraction of cells from CD activity
group 2 within the BBox.

Figure 3: This figure illustrates the step-by-step workflow of the whole study
data cohort exploration process, showcasing the rigorous methodologies
employed in image analysis, cell classification, nearest neighbor calculation,
visualization, and ultimate pattern recognition using a graph-based nearest
neighbor signature approach.

##  3 Experiments and Results

###  3.1 Dataset

CD can manifest anywhere in the gastrointestinal tract. In our study, we
applied our proposed workflow to two datasets acquired from different
institutions, with a specific focus on the anatomical region of the rectum.
The first dataset was obtained from the Emory University School of Medicine
(Emory), also in the Gut Cell Atlas Crohn’s Disease Consortium, and includes 8
biopsies from children. Among these, 4 biopsies are from a healthy control
group, while the remaining 4 biopsies are classified as CD active.

The second dataset comprises 143 biopsies stained with H&E, obtained in a
deidentified form from Vanderbilt University Medical Center (VUMC) under
Institutional Review Board (IRB) approval, specifically Vanderbilt IRB #191738
and #191777 [15]. All biopsies were collected from adult CD patients and
scored by a single pathologist, resulting in 97 biopsies classified as normal
and 46 biopsies marked as active, categorized into subcategories of mild,
moderate, and severe. It is worth noting that the VUMC dataset lacks a healthy
control group. Thus, for a “normal” comparator group, the term “CD normal”
refers to patients diagnosed with CD, but the pathologic review of collected
tissues were normal, and showed no acute or chronic changes on pathology due
to medical therapies.

###  3.2 Experiment design

For each cell type category, which includes neutrophils, epithelial cells,
lymphocytes, plasma cells, eosinophils, and connective tissue, we conduct
t-SNE visualization and compute the odds ratios for the research institutes
separately. Table 1 presents a summary of the data points for each site and
the six cell subtype categories. To explore the potential ROIs, we investigate
them in both the t-SNE scatter plots, where we identify exclusive regions
using BBox, and the t-SNE KDE contour plots, where our attention is on areas
with relatively high probabilities. All experiments were executed on a
workstation boasting 96 CPU cores, 250 GiB of RAM, and an NVIDIA RTX A6000
GPU.

Table 1: Data summary and cell nuclei type counts across different institutions and patient groups. |  | Institute 1: VUMC | Institute 2: Emory
---|---|---|---
|  | CD normal | CD active | Healthy control | CD active
Count | Sample biopsies | 97 | 46 | 4 | 4
Neutrophils (neu) | 16,424 | 33,103 | 384 | 1,022
Epithelial cell (epi) | 5,528,718 | 2,409,172 | 50,479 | 209,950
Lymphocytes (lym) | 4,037,761 | 2,946,595 | 37,326 | 379,809
Plasma cell (pla) | 738,568 | 607,164 | 9,651 | 21,486
Eosinophils (eon) | 50,173 | 61,557 | 1,148 | 1,437
Connective tissue (con) | 1,406,897 | 887,472 | 17,838 | 60,172

###  3.3 Results

We provide the outcomes of spatial pattern analysis pertaining to individual
cell types across the two institutes as follows.

Neutrophils (Figure 4). The VUMC dataset reveals that CD active tissues tend
to exhibit a higher count of 10-NN shapes containing neutrophils and
lymphocytes, as well as plasma, with a moderate amount of connective tissue.
Moreover, it indicates that when lymphocytes dominate the surroundings without
epithelial cells, CD active occurrences surpass those in CD normal tissue. CD
normal tissues, on the other hand, tend to have a greater involvement of
epithelial cells. In the Emory dataset, it is evident that lymphocytes play a
crucial role in the 10-NN interaction with neutrophils. Interestingly, even
when both lymphocytes and epithelial cells are dominant, the prevalence
remains higher in CD active cases. Furthermore, the dominance of lymphocytes
and plasma is associated with a high occurrence rate of healthy controls.

Eosinophils (Figure 5). In the VUMC dataset, there is a notable significance
of lymphocytes and plasma in CD active tissues, whereas eosinophils in CD
normal tissues tend to exhibit a higher tendency towards involvement with
epithelial and plasma components. However, the Emory dataset reveals a
contrasting pattern.

Connective (Figure 6). In the VUMC dataset, shape patterns between CD active
and CD normal tissues are remarkably similar. CD active tissues show moderate
elevation in lymphocyte and connective tissue presence, while CD normal
tissues display increased involvement of epithelial elements. Conversely, the
Emory dataset presents greater diversity. It mirrors the VUMC CD active
pattern; moreover, dominant connective tissue aligns with healthy controls,
and higher plasma presence associates with the control group.

Plasma (Figure 7). No distinct pattern is identified that is specific to CD
active tissues. In the case of CD normal tissues, there is a tendency for an
even distribution of involvement among epithelial, lymphocyte, plasma, and
connective tissue. This pattern is similarly observed in the Emory dataset.
However, when the surroundings solely consist of lymphocytes and plasma, this
combination appears more frequently in CD active cases.

Plasma (Figure 8). In the VUMC dataset, no notable pattern difference in
lymphocyte distribution between CD active and CD normal tissues is discerned,
where lymphocytes dominate the 10-NN arrangement. This similarity in pattern
is also observed in the Emory dataset for CD active cases. However, within the
Emory dataset, healthy control cases exhibit a more varied pattern, with the
additional presence of plasma and connective tissue in the local environment,
alongside lymphocytes.

Epithelial (Figure 9). Both datasets from the different institutions
consistently indicate that epithelial cells predominantly interact with other
epithelial cells, regardless of whether the tissue is CD active, CD normal, or
from a healthy control.

Figure 4: Visualization and quantification of neutrophils surrounding a 10-NN
shape pattern recognition. Figure 5: Visualization and quantification of
eosinophils surrounding a 10-NN shape pattern recognition.

##  4 Discussion and Conclusion

In this paper, we delve into the exploration of a broader morphometric aspect
within the local arrangement of cells, shifting the focus from neutrophil
counting to the realm of CD investigation. Our proposed approach introduces a
graph-based signature that is customized to portray a distinct local
arrangement for each cell. Additionally, we have meticulously designed a
comprehensive visualization workflow that illuminates the intricate
interconnections among signature patterns within both the CD active patient
and CD normal cohorts among adults, as well as between the CD active patient
cohort and healthy controls among children. Our study encompasses
investigations carried out across two distinct research institutes, centered
around the rectum region. The diversified conclusions stemming from shape
patterns, extending beyond neutrophils, can be attributed to the diverse array
of data acquisition methods. For instance, the VUMC dataset primarily
encompasses adult samples and lacks healthy patient tissue, whereas the
control group comprises CD biopsies classified as normal. In contrast, the
Emory dataset is sourced from pediatric samples, introducing an additional
layer of variation in the data collection process.

In conclusion, our findings indicate that cells do not consistently co-locate
in the same manner between CD active and CD normal cohorts for adults rectum
dataset, or between CD active and healthy control for children. To delve
deeper into these distinct shape patterns, we can leverage RNA-seq to identify
cell types, unveil gene expression patterns, facilitate biomarker discovery,
enable spatial mapping, and uncover disease mechanisms. Moreover, integrating
morphological spatial features, such as considering actual cell distances in
the neighborhood of the spatial signature, could potentially enhance our
analysis. The observed disparities in spatial distributions across the two
research institutes underscore the significance of collaborative endeavors
among healthcare institutions.

##  5 ACKNOWLEDGMENTS

This research was supported by The Leona M. and Harry B. Helmsley Charitable
Trust grant G-1903-03793 and G-2103-05128, NSF CAREER 1452485, NSF 2040462,
and in part using the resources of the Advanced Computing Center for Research
and Education (ACCRE) at Vanderbilt University, Nashville, TN. This project
was supported in part by the National Center for Research Resources, Grant UL1
RR024975-01, and is now at the National Center for Advancing Translational
Sciences, Grant 2 UL1 TR000445-06, the National Institute of Diabetes and
Digestive and Kidney Diseases, the Department of Veterans Affairs
I01BX004366,I01CX002171, and I01CX002573. The de-identified imaging dataset(s)
used for the analysis described were obtained from ImageVU, a research
resource supported by the VICTR CTSA award (ULTR000445 from NCATS/NIH),
Vanderbilt University Medical Center institutional funding and Patient-
Centered Outcomes Research Institute (PCORI; contract CDRN-1306-04869). This
work is supported by NIH grants T32GM007347, R01DK135597, and R01DK103831. We
extend gratitude to NVIDIA for their support by means of the NVIDIA hardware
grant. ChatGPT was utilized for proofreading and grammar checking, and the
results were validated by human review to ensure accuracy of facts and intent.

##  Appendix A Supplementary information on spatial pattern analysis

Figure 6: Visualization and quantification of connective tissue surrounding a
10-NN shape pattern recognition. Figure 7: Visualization and quantification of
plasma cells surrounding a 10-NN shape pattern recognition. Figure 8:
Visualization and quantification of lymphocytes surrounding a 10-NN shape
pattern recognition. Figure 9: Visualization and quantification of epithelial
cells surrounding a 10-NN shape pattern recognition.

## References

  * [1] Baumgart, D. C. and Sandborn, W. J., “Crohn’s disease,” The Lancet 380(9853), 1590–1605 (2012).
  * [2] Hamdeh, S., Aziz, M., Altayar, O., Olyaee, M., Murad, M. H., and Hanauer, S. B., “Early vs late use of anti-tnfa therapy in adult patients with crohn disease: a systematic review and meta-analysis,” Inflammatory bowel diseases 26(12), 1808–1818 (2020).
  * [3] Failmezger, H., Muralidhar, S., Rullan, A., de Andrea, C. E., Sahai, E., and Yuan, Y., “Topological tumor graphs: a graph-based spatial model to infer stromal recruitment for immunosuppression in melanoma histology,” Cancer research 80(5), 1199–1209 (2020).
  * [4] Xu, H., Cong, F., and Hwang, T. H., “Machine learning and artificial intelligence–driven spatial analysis of the tumor immune microenvironment in pathology slides,” European Urology Focus 7(4), 706–709 (2021).
  * [5] Xu, H., Cha, Y. J., Clemenceau, J. R., Choi, J., Lee, S. H., Kang, J., and Hwang, T. H., “Spatial analysis of tumor-infiltrating lymphocytes in histological sections using deep learning techniques predicts survival in colorectal carcinoma,” The Journal of Pathology: Clinical Research 8(4), 327–339 (2022).
  * [6] Saltz, J., Gupta, R., Hou, L., Kurc, T., Singh, P., Nguyen, V., Samaras, D., Shroyer, K. R., Zhao, T., Batiste, R., et al., “Spatial organization and molecular correlation of tumor-infiltrating lymphocytes using deep learning on pathology images,” Cell reports 23(1), 181–193 (2018).
  * [7] Pai, R. K., Lauwers, G. Y., and Pai, R. K., “Measuring histologic activity in inflammatory bowel disease: why and how,” Advances in anatomic pathology 29(1), 37–47 (2022).
  * [8] Gao, S.-Q., Huang, L.-D., Dai, R.-J., Chen, D.-D., Hu, W.-J., and Shan, Y.-F., “Neutrophil-lymphocyte ratio: a controversial marker in predicting crohn’s disease severity,” International journal of clinical and experimental pathology 8(11), 14779 (2015).
  * [9] Graham, S., Vu, Q. D., Raza, S. E. A., Azam, A., Tsang, Y. W., Kwak, J. T., and Rajpoot, N., “Hover-net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images,” Medical image analysis 58, 101563 (2019).
  * [10] Graham, S., Jahanifar, M., Vu, Q. D., Hadjigeorghiou, G., Leech, T., Snead, D., Raza, S. E. A., Minhas, F., and Rajpoot, N., “Conic: Colon nuclei identification and counting challenge 2022,” arXiv preprint arXiv:2111.14485 (2021).
  * [11] Lu, M. Y., Williamson, D. F., Chen, T. Y., Chen, R. J., Barbieri, M., and Mahmood, F., “Data-efficient and weakly supervised computational pathology on whole-slide images,” Nature Biomedical Engineering 5(6), 555–570 (2021).
  * [12] Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al., “Scikit-learn: Machine learning in python,” the Journal of machine Learning research 12, 2825–2830 (2011).
  * [13] De Berg, M., [Computational geometry: algorithms and applications ], Springer Science & Business Media (2000).
  * [14] Van der Maaten, L. and Hinton, G., “Visualizing data using t-sne.,” Journal of machine learning research 9(11) (2008).
  * [15] Bao, S., Chiron, S., Tang, Y., Heiser, C. N., Southard-Smith, A. N., Lee, H. H., Ramirez, M. A., Huo, Y., Washington, M. K., Scoville, E. A., et al., “A cross-platform informatics system for the gut cell atlas: integrating clinical, anatomical and histological data,” in [Medical Imaging 2021: Imaging Informatics for Healthcare, Research, and Applications ], 11601, 8–15, SPIE (2021).