# Unlocking Large Language Model’s Planning Capabilities with Maximum
Diversity Fine-tuning

Wenjun Li
Singapore Management University
<EMAIL_ADDRESS>
&Changyu Chen
Singapore Management University
<EMAIL_ADDRESS>
&Pradeep Varakantham
Singapore Management University
<EMAIL_ADDRESS>

###### Abstract

Large language models (LLMs) have demonstrated impressive task-solving
capabilities, achieved through either prompting techniques or system designs.
However, concerns have arisen regarding their proficiency in planning tasks,
as they often struggle to generate valid plans. This paper investigates the
impact of fine-tuning on LLMs’ planning capabilities. Our findings indicate
that LLMs can achieve good performance in planning through substantial
(thousands of specific examples) fine-tuning. However, fine-tuning is
associated with significant economic and computational costs. To address this
challenge, we propose the Maximum Diversity Fine-Tuning (MDFT) strategy to
improve the sample efficiency of fine-tuning in the planning domain.
Specifically, our algorithm, referred to as MDFT-g, encodes the planning task
instances with their graph representations and selects a subset of samples in
the vector space that maximizes data diversity. We empirically demonstrate
that MDFT-g consistently outperforms existing baselines at various scales
across multiple benchmark domains.

##  1 Introduction

Large language models (LLMs) have demonstrated impressive capabilities in a
variety of applications beyond traditional natural language processing (NLP)
tasks. For instance, they can accomplish complex tasks involving planning in
environments like Minecraft [21, 25] and household settings [10, 23], and
solve math problems using reasoning [22, 24]. Despite these impressive
capabilities, concerns have been raised about their proficiency in planning
tasks. Numerous studies [19, 14, 11, 7] have shown that LLMs, on their own,
often struggle to generate valid plans. As a result, enhancing the planning
abilities of LLMs has emerged as a prominent research area.

In this paper, we focus on enhancing the planning capabilities of Large
Language Models (LLMs) within rigorous planning settings, often referred to as
"automated planning". Despite extensive criticism, there has been little
concrete progress in fundamentally improving the planning capabilities of LLMs
within the field of automated planning. In this paper, we demonstrate that
fine-tuning remains the most effective approach to tackle this challenge,
achieving good planning capabilities with solved rates exceeding 95%.

Many works [5, 17, 4] have demonstrated that fine-tuning can significantly
enhance LLM’s task-specific performance. However, comprehensive research on
fine-tuning in the planning domain remains limited. To address this gap, we
provide an in-depth analysis of how fine-tuning affects LLMs’ planning
capabilities. Specifically, we investigate the impact of data scaling and
diversity on the fine-tuning outcomes, focusing on both planning capabilities
and generalization performance of LLMs. Surprisingly, we found that
substantial fine-tuning significantly improves LLM’s planning capabilities to
a satisfactory level, which contradicts the existing prejudice that LLMs are
weak at planning. In summary, our research addresses two main questions: (1)
How does fine-tuning impact the planning capabilities of LLMs? (2) Can we
identify the most effective samples to enhance sample efficiency in fine-
tuning?

Fine-tuning both closed-source models like OpenAI’s GPT-x [2] and open-source
models such as Meta’s Llama-x [18] entails substantial economic and
computational costs. For example, fine-tuning GPT-3.5 once with just 5000
planning task examples in the Blocksworld domain costs approximately 100 USD.
Additionally, fine-tuning Llama-3-8b requires high-end and expensive GPUs
(e.g., A100, H100) and a significant amount of time. Given these constraints,
our goal is to enhance the planning capabilities of LLMs while minimizing the
required fine-tuning data. In this paper, we explore the impact of both the
quantity and diversity of fine-tuning data on training outcomes. We propose
the Maximum Diversity Fine-Tuning (MDFT) strategy to improve sample efficiency
in the planning domain. Our algorithm, referred to as MDFT-g, encodes planning
instances in vector space using their graph representations and selects the
most diverse and representative examples. In our experiments, we empirically
demonstrate that MDFT-g consistently outperforms existing baseline methods at
the same scale.

Figure 1: Algorithm overview. (a) Training samples in natural language. (2)
Samples encoded in vector space. (3) A subset of samples selected by MDFT-g.
The red stars are the selected ones. (d) Fine-tune LLMs with the subset of
samples.

Overall, our work makes two key contributions. First, we conducted a
comprehensive investigation into how data scaling and diversity affect the
fine-tuning outcomes of LLMs, particularly in the planning domain. Second, we
introduced MDFT-g, a novel approach that leverages graph representations to
encode planning instances and selects the most diverse and representative
samples. Our empirical results demonstrate that MDFT-g significantly improves
sample efficiency and outperforms existing baseline methods.

##  2 Related Work

Enhancing LLM’s planning capabilities with prompting. Prompt controls the
input to an LLM using a template or a cue to elicit the desired output.
Approaches like Chian of Thoughts (CoT, [22]) and Tree of Thoughts (ToT, [24])
enable complex reasoning and planning capabilities through intermediate
reasoning steps. However, these prompting techniques have shown limited
effectiveness in improving performance in automated planning domains. For
example, [19] showed that CoT prompting resulted in only a 1% improvement in
the Blocksworld domain, with GPT-4 [1] achieving a 34.6% solved rate without
CoT prompting and 35.6% with it.

LLMs with access to external tools. Due to LLMs’ tendency to generate
responses with hallucinations and their lack of strict adherence to actions
and predicates, some approaches integrate external tools to provide feedback.
For example, [11] and [7] utilize the Planning Domain Definition Language
(PDDL) executor to verify the validity of generated plans and provide feedback
(such as explanations for why plans are not executable) to aid LLMs in
reflection and plan refinement. However, these approaches do not fundamentally
enhance the inherent planning capabilities of LLMs and require complex
integration engineering work. Additionally, the effectiveness of the
traditional planner relies heavily on the quality of the translation from
natural language to PDDL. Any inaccuracies or omissions in this translation
can result in incorrect plans or failed validations.

Enhancing LLM’s task-specific capabilities with fine-tuning. Fine-tuning
refers to the process of updating the parameters of a pre-trained model using
task-specific data. Initial studies, such as Less Is More for Alignment (LIMA,
[27]), have investigated the effects of training data quantity and quality on
fine-tuning outcomes in typical natural language processing tasks like Q&A and
creative writing. However, to the best of our knowledge, there has been no
comprehensive study on how fine-tuning data (considering aspects like
quantity, diversity, composition, etc.) affects the planning capabilities of
LLMs. This paper pioneers the exploration of these research questions.

##  3 Approach: Maximum Diversity Fine-tuning

We choose supervised fine-tuning (SFT) as our fine-tuning approach in this
paper. SFT data consists of two components: queries and the associated
responses. Previous studies [27, 26] have shown that higher quality responses
and more diverse queries lead to better LLM performance after fine-tuning. In
automated planning domains, we can employ traditional planning solvers (e.g.,
Fast Downward [8]) to find the optimal plans and achieve the highest quality
in responses. As a result, how to capture the diversity of the fine-tuning
data becomes the main challenge.

Unlike [27, 26] that utilized human resources to capture the diversity in
queries, we aim to automatically identify the diverse and effective samples.
Moreover, given the high economic and computational costs associated with
fine-tuning, achieving higher sample efficiency is crucial. To address these,
we propose utilizing the Maximum Diversity Fine-Tuning (MDFT) strategy to
enhance the planning capabilities of LLMs using minimal data. Maximizing
diversity within a subset of samples is known as the Maximum Diversity Problem
(MDP, [6, 13]) in the literature, which has been studied for decades. MDP is
NP-hard, meaning that finding the exact optimal solution is computationally
challenging for large datasets. However, several heuristic and approximation
algorithms exist to find good solutions. In this paper, we employ clustering-
based approaches to identify representative points, thereby ensuring diversity
within the dataset.

###  3.1 Representing Planning Tasks as Graphs

MDP algorithms operate in numeric and vector spaces, so the first step is to
convert planning tasks from natural language into vector space.

Natural language can be easily and efficiently encoded into vector
representations using language embedding tools such as the Universal Sentence
Encoder [3] and Self-BLEU [28]. However, due to the narrative nature of
planning tasks, planning tasks with distinct solutions can only differ in a
few words in the natural language descriptions and result in highly similar
language embeddings. For example, in Figure 2(a), switching the positions of
the orange and green blocks in the initial configuration while keeping other
conditions constant changes the task descriptions slightly. The original task
description is, "As initial conditions, I have that the orange block is clear,
the orange block is on top of the green block, …", whereas the altered task
description is, "As initial conditions, I have that the green block is clear,
the green block is on top of the orange block, …". Despite having different
solutions, these two tasks produce very similar language embeddings that are
close in vector space. This issue makes subsequent diversity maximization
difficult and inaccurate. Additional results and analyses on the limitations
of language embeddings can be found in the appendix.

Therefore, a more representative embedding approach specifically tailored for
planning tasks is necessary to capture the nuances between tasks. Inspired by
[15, 16], we use graph representations to encode planning tasks. Each planning
task consists of an initial configuration of objects and a goal configuration
of objects. To solve the task, the LLM needs to perform sequential actions to
transform the initial configuration into the goal configuration. In graph
representation, each object is a node, and each predicate is a (directed) edge
between two nodes. An example task from the Blocksworld domain and its graph
representation are shown in Figure 2. Once the planning tasks are converted
into graphs, we can efficiently encode these graphs into vector embeddings. To
fully capture the information of a task, we concatenate the vectors of the
initial configuration and the goal configuration into a single graph
representation. Additional details are provided in the appendix.

Figure 2: (a) An example task from Blocksworld. Left: initial configuration.
Right: goal configuration. (b) A graph representation of the example task,
where nodes are objects and edges are predicates.

###  3.2 Selecting Maximum Diversity Data

Instead of solely maximizing diversity among the subset of samples, we also
want the selected samples to be representative to enhance the effectiveness of
downstream fine-tuning. To achieve this, we adopt a straightforward heuristic
using the clustering-based method to select the maximum diversity fine-tuning
data in vector space.

Assume we have a total number of $N$ tasks and we want to find a subset of $k$
tasks that maximize the overall diversity. The diversity is defined as the sum
of all pair-wise distances between the $k$ samples. To make the clustering
more effective, we first perform dimension reduction (e.g., t-SNE [20]) on the
embedding space. Subsequently, we perform k-means clustering on the entire set
of data points, with $k$ used as the number of clusters. For each cluster, we
find the point closest to the cluster centroid, select it to represent the
cluster, and include it in the final subset.

The clustering-based method is compatible with any embedding approaches that
can convert planning tasks from natural language to vector representations.
Note that the clustering requires a suitable distance metric. For instance, we
employed L2 norm distance for language embeddings and edit distance for graph
embeddings in our experiments. We denote the maximum diversity fine-tuning
data selection with language embeddings as MDFT-l and graph embeddings as
MDFT-g, respectively. The complete procedures of MDFT-l and MDFT-g are
presented in Algorithm 1.

Input: $N$ tasks, subset size $k$, embedding method $f(\cdot)$, distance
metric $c(\cdot,\cdot)$

1 Initialize empty subset $\mathcal{D}$, embeddings set $\mathbb{E}$

2 for  _$i=1$ to $N$_ do

3 Encode task $t_{i}$ as vector embedding, $e_{i}=f(t_{i})$

4 Insert $e_{i}$ into $\mathbb{E}$

5 end for

6Perform dimension reduction on the embedding space

7 Perform k-means clustering with distance metric $c(\cdot,\cdot)$ over the
embeddings with $k$ clusters

8 Obtain $k$ centroids of the clusters

9 for  _$i=1$ to $k$_ do

10 In each cluster, find the $e_{i}$ closest to the cluster centroid

11 Retrieve the task instance associated with $e_{i}$ and add it to
$\mathcal{D}$

12

13 end for

Output: A subset of tasks, $\mathcal{D}$

Algorithm 1 Selecting Maximum Diversity Fine-tuning Data

##  4 Experiment Setup

Benchmark Domains: We utilize two widely adopted benchmark domains for
automated planning from the International Planning Competition (IPC, [12]).
Detailed information about domain properties, actions, predicates, dataset
generation, prompts, and other specifics can be found in the appendix.

Blocksworld: This domain comprises a set of blocks, a table, and a robot hand.
Blocks can be placed on other blocks or on the table. A block with nothing on
top of it is considered clear, and the robot hand can hold one block or be
empty. The robot hand can perform four actions: pick-up, put-down, stack, and
unstack. The goal is to find a plan to transition from the initial
configuration of blocks to the goal configuration. Blocks are uniquely
identified by their colors. The complexity of Blocksworld tasks varies mainly
with the number of blocks. We randomly created thousands of instances with
different numbers of blocks and used a planning solver [8] to find the optimal
plans. Note that the testing tasks are separated from the training tasks to
ensure that the LLMs never see the testing tasks during fine-tuning. Detailed
experimental settings are provided with each table and figure. More details
about the Blocksworld dataset can be found in the appendix.

Logistics: The objective in this domain is to transport packages to specified
goal locations. Locations are grouped by cities. Trucks move packages within a
city, and airplanes move packages between cities. Each city has one truck and
an airport. There can be multiple airplanes and packages located in different
cities and locations. The Logistics domain is much more complex than
Blocksworld and typically requires longer plans. We generated six thousand
instances with varying numbers of cities, locations, packages, and airplanes.

Base Models: We selected GPT-3.5-turbo-0125 and Llama-3-8b as the base models
for addressing Research Question 1 because they are the latest and most
powerful closed-source and open-source LLMs, respectively. GPT-4 [1] was
omitted from our experiments due to its limited experimental access and lack
of public availability for fine-tuning. Additionally, we also conducted
experiments on Llama-2-7b [18] for Research Question 2.

Baselines: We compare the proposed MDFT-g algorithm with two baselines: Random
and MDFT-l. The Random baseline uniformly samples $k$ instances from the
entire training dataset. MDFT-l differs from MDFT-g only in the embedding
method and distance metric, as described in the Approach section and Algorithm
1.

Evaluation: In our experiments, LLMs respond in natural language, and the
generated plans are translated to PDDL and then verified using VAL [9]. A plan
is considered correct as long as it successfully transitions from the initial
configuration to the goal configuration. An analysis of the optimality rate of
the generated plans is provided in the appendix.

Prompt: Our experiments use two types of prompts: one-shot and zero-shot
prompts. The one-shot prompt includes three components: domain instructions,
an example task with its solution, and a query task. The zero-shot prompt
contains only the domain instructions and the query task. Further details
about the prompt settings are available in the appendix. In our experiments,
we found that fine-tuned models perform better with zero-shot prompts, while
un-fine-tuned models perform better with one-shot prompts. This is because the
fine-tuned models have seen many example responses during fine-tuning,
allowing them to follow the desired format and provide more concise responses.
Consequently, we evaluate un-fine-tuned models with one-shot prompts and fine-
tuned models with zero-shot prompts.

##  5 Experiment Results

###  5.1 Research Question 1. How does fine-tuning impact the planning
capabilities of LLMs?

To evaluate the planning capabilities and transferability of LLMs precisely,
we prepared multiple testing tasks with varying numbers of blocks in the
Blocksworld domain. Specifically, there are 100, 1000, 500, and 500 testing
tasks for 3-block, 4-block, 5-block, and 6-block settings, respectively. In
contrast, the Logistics domain has too many variables (e.g., the number of
cities, locations, trucks, airplanes, etc.) that affect task complexity,
making it challenging to identify which variable impacts task difficulty the
most. Therefore, we randomly selected 300 instances with varying numbers of
variables as the testing data for the Logistics domain.

Table 1: Task solved rate of un-finetuned LLMs. Domain | GPT-3.5-turbo | Llama-3-8b
---|---|---
Blocksworld | 3-block | 8.0% (8/100) | 0.0% (0/100)
4-block | 3.7% (37/1000) | 0.0% (0/1000)
5-block | 1.2% (6/500) | 0.0% (0/500)
6-block | 0.4% (2/500) | 0.0% (0/500)
Logistics |  | 0.7% (2/300) | 0.0% (0/300)

####  5.1.1 Significant Improvement Brought by Fine-tuning

First, we measured the planning capabilities of LLMs before fine-tuning, with
the results presented in Table 1. Notably, LLMs without fine-tuning exhibit
weak planning abilities when provided with only one example in the prompt.
These results align with existing criticisms of LLMs’ planning capabilities in
the literature.

Next, we evaluated the planning capabilities of LLMs after fine-tuning them on
planning tasks using two sample sizes: large (1000 samples) and small (100
samples). Table 2 presents the test performance of LLMs after fine-tuning.
Remarkably, the planning capabilities of LLMs were significantly enhanced
after fine-tuning, even with a small number of samples. Additionally, we
tested the fine-tuned models on the dataset from [19], where GPT-3.5-turbo and
Llama-3-8b achieved solved rates of 90.6% and 84.3%, respectively. Humans are
reported in [19] to have a solved rate of 78%. These findings challenge the
prevailing notion that LLMs are inherently weak at planning.

Table 2:  Task solved rate of fine-tuned LLMs. GPT-3.5-turbo and Llama-3-8b fine-tuned with 100 and 1000 samples are denoted as GPT-3.5-100/Llama-3-100 and GPT-3.5-1k/Llama-3-1k, respectively. We use 4-block samples for Blocksworld and samples with varying numbers of variables for Logistics. Domain | GPT-3.5-100 | GPT-3.5-1k | Llama-3-100 | Llama-3-1k
---|---|---|---|---
Blocksworld | 3-block | 65.0% | 98.0% | 57.0% | 85.0%
4-block | 61.8% | 91.9% | 43.9% | 91.3%
5-block | 36.4% | 65.8% | 19.6% | 46.4%
6-block | 15.4% | 34.0% | 3.6% | 15.0%
Logistics |  | 7.5% | 73.0% | 2.8% | 72.7%

####  5.1.2 Data Scaling Effect

We then examined the effect of data scaling on LLMs’ planning capabilities.
Figure 3 shows LLM’s planning capabilities improve as the amount of fine-
tuning data increases. However, this improvement follows an asymptotic pattern
– while initial increases in sample size lead to substantial performance
gains, the rate of improvement (slope) diminishes as more samples are added.
When provided with 4000 examples, LLMs achieve excellent performance levels.
In Blocksworld, GPT-3.5-turbo and Llama-3-8b achieved 96.5% and 99.3% solved
rates, respectively, with 4000 fine-tuning samples. In the Logistics domain,
Llama-3-8b reached a 94.7% solved rate with 4000 fine-tuning samples.

Figure 3: Model performance versus the number of fine-tuning samples.
GPT-3.5-turbo finetuned with 4000 Logistics samples is omitted in this
experiment due to high economic costs.

From the above results, we can conclude that GPT-3.5-turbo demonstrates
superior planning capabilities compared to the Llama models both before and
after fine-tuning. However, GPT-3.5-turbo is outperformed by Llama-3-8b when
both are fine-tuned with 4000 samples in our experiment. This discrepancy is
because we restricted the number of fine-tuning epochs for GPT-3.5-turbo due
to high economic costs, whereas the Llama models were fine-tuned for
significantly more epochs.

####  5.1.3 Transferability

Furthermore, we explored the transfer performance of LLMs to different tasks
and domains after fine-tuning.

In-domain Transferability: We assessed LLM’s transfer performance to different
tasks within the same domain after fine-tuning. The base models, GPT-3.5-turbo
and Llama-3-8b, were fine-tuned with 1000 samples of 4-block, 5-block, and
6-block tasks, respectively. The fine-tuned models were then evaluated on
tasks with different block numbers than those used in fine-tuning. Note that
we did not fine-tune the base models with 1000 3-block samples because the
3-block setting does not have enough distinct samples. The in-domain
transferability results are provided in Table 3.

Our results indicate that LLMs exhibit good in-domain transferability and can
apply their acquired planning capabilities to new challenges. For example,
GPT-3.5 fine-tuned with 5-block tasks transfers well (68.0%) to 6-block tasks,
even when the tasks involve unseen block colors. Notably, LLMs transfer better
to less complex tasks within the same domain. GPT and Llama fine-tuned with
6-block tasks did not achieve the best performance and transferability because
1000 samples are insufficient for LLMs to achieve a good performance on
6-block tasks.

Table 3: LLM’s transferability within the same domain after fine-tuning. GPT-3.5 and Llama-3 finetuned with x-block tasks are denoted as GPT-3.5-$x$block and Llama-3-$x$block respectively. | Testing Tasks
---|---
Finetuned Model | 3-block | 4-block | 5-block | 6-block
GPT-3.5-4block | 98.0% | 91.9% | 65.8% | 34.0%
GPT-3.5-5block | 81.0% | 88.0% | 84.8% | 68.0%
GPT-3.5-6block | 74.0% | 75.4% | 77.0% | 70.6%
Llama-3-4block | 85.0% | 91.3% | 46.4% | 15.0%
Llama-3-5block | 73.0% | 82.3% | 71.6% | 39.2%
Llama-3-6block | 63.0% | 65.6% | 69.2% | 58.6%

Cross-domain Transferability: To assess cross-domain transferability, we fine-
tuned the base models in one domain and then evaluated them in another.
Specifically, we fine-tuned the base models with 200 tasks from the
Blocksworld domain and evaluated their performance in the Logistics domain.
Similarly, we fine-tuned the base models with 400 tasks from the Logistics
domain and then evaluated them in the Blocksworld domain. We used relatively
small sample sizes in this experiment to prevent the LLMs from overfitting to
the response patterns and dynamics of a single domain. The cross-domain
transferability results are presented in Table 4.

Table 4: LLM’s transferability across domains. |  |  | Fine-tuning Tasks
---|---|---|---
Test Domain | Model | un-finetuned | Blocksworld | Logistics
Blocksworld | GPT-3.5 | 3.7% | 72.4% | 3.1%
Llama-3 | 0% | 54.2% | 0.3%
Logistics | GPT-3.5 | 0.7% | 0.3% | 45.8%
Llama-3 | 0% | 0% | 44.4%

Notably, LLMs exhibit poor transferability across different planning domains
after fine-tuning, even worse than un-fine-tuned models. This is due to the
significant differences between automated planning domains in terms of
actions, predicates, objects, etc., making it difficult for LLMs to generalize
across them. Consequently, to enable LLMs to acquire multi-task capabilities,
it is necessary to fine-tune them on tasks from multiple domains collectively.
A detailed analysis of how data composition affects fine-tuning outcomes is
provided in the extended experiment section in the appendix. In summary, while
fine-tuning on mixed data enables LLMs to effectively acquire multi-task
planning capabilities, it also introduces a slightly higher variance in
outcomes.

###  5.2 Research Question 2: Can we identify the most effective samples to
enhance sample efficiency in fine-tuning?

In this section, we demonstrate that MDFT-g identifies the most representative
and effective samples, thereby improving the sample efficiency in fine-tuning.
Additionally, we found that MDFT-l also outperforms Random, highlighting the
value of the maximum diversity fine-tuning (MDFT) strategy. We simplified
testing data composition for Blocksworld, as detailed performance on different
types of tasks was unnecessary for analysis in this section. We uniformly
sampled from the four types of testing settings (i.e., 3-block, 4-block,
5-block, 6-block) to collectively create a testing dataset with 1000
instances. Testing data for Logistics remains consistent with the previous
sections.

First, we illustrate the effectiveness of the clustering-based method in
solving the maximum diversity problem. Figure 4 shows the subsets of samples
identified by MDFT-g and MDFT-l. In both plots, the selected samples (red
stars) are located nearest to the centroids of each cluster, ensuring a
diverse and broad coverage of the entire vector space. However, the data
points in each cluster for MDFT-l (b) are very dense and close together due to
the limitations of language embeddings in capturing the complexity of planning
tasks. This results in clusters being far apart, influenced more by narrative
differences than by task structure. In contrast, MDFT-g (a), which uses graph
embeddings, represents tasks with greater expressiveness. This method captures
the structural nuances of planning tasks more effectively, resulting in more
representative clustering and diverse sample selection.

Figure 4: Illustration of subset samples selected by (a) MDFT-g and (b) MDFT-l. Each point represents a planning task in Blocksworld, encoded in vector space. The sample point closest to each cluster centroid is selected for the subset. Table 5: Task solved rate in the Blocksworld domain. Results are presented as mean $\pm$ standard deviation. GPT and Llama results are collected across three and five random seeds respectively. |  | Subset Size
---|---|---
Base Model | Algorithm | $k=100$ | $k=200$ | $k=400$ | $k=1000$
GPT-3.5-turbo | Random | 61.8$\pm$1.9% | 72.4$\pm$1.8% | 81.9$\pm$1.5% | 91.9$\pm$1.2%
MDFT-l | 60.4$\pm$1.6% | 73.2$\pm$1.4% | 82.8$\pm$1.3% | 92.5$\pm$1.0%
MDFT-g | 71.7$\pm$1.7% | 77.6$\pm$2.0% | 85.8$\pm$1.3% | 94.9$\pm$0.8%
Llama-2-7b | Random | 28.8$\pm$3.6% | 30.3$\pm$8.6% | 50.0$\pm$1.0% | 74.6$\pm$3.7%
MDFT-l | 24.8$\pm$4.4% | 32.6$\pm$3.8% | 51.5$\pm$1.4% | 74.2$\pm$2.9%
MDFT-g | 32.5$\pm$2.1% | 37.2$\pm$4.5% | 52.9$\pm$1.3% | 75.7$\pm$0.7%
Llama-3-8b | Random | 43.9$\pm$2.9% | 54.2$\pm$6.8% | 67.6$\pm$3.3% | 86.3$\pm$0.8%
MDFT-l | 42.0$\pm$3.9% | 53.6$\pm$5.4% | 69.2$\pm$4.5% | 87.1$\pm$0.7%
MDFT-g | 51.2$\pm$5.3% | 59.5$\pm$3.7% | 71.6$\pm$2.4% | 88.5$\pm$0.7%

Next, we empirically demonstrate MDFT-g’s effectiveness at different scales.
In Blocksworld, we created five thousand instances with varying numbers of
blocks as the training dataset. Using Random, MDFT-l, and MDFT-g, we selected
task subsets of varying sizes ($k=$100, 200, 400, and 1000) and fine-tuned the
base models, then evaluated them on testing data. Complete results for
Blocksworld and Logistics are shown in Tables 5 and 6. MDFT-g consistently
outperforms Random and MDFT-l at various scales in both domains. MDFT-g
significantly improves sample efficiency in fine-tuning, particularly at
smaller scales. For instance, MDFT-g with $k=100$ achieves a 71.7% solved
rate, a performance that requires Random and MDFT-l to have around 200
samples. MDFT-g shows smaller performance margins as sample size increases
because the improvement brought by a new sample diminishes as sample size
increases. Thus, MDFT-g is particularly effective and beneficial in scenarios
where fine-tuning incurs high economic or computational costs, making it ideal
for applications with limited resources or stringent efficiency requirements.

Table 6: Task solved rate in Logistics. GPT and Llama results are collected across three and five random seeds respectively. GPT fine-tuned with 1000 samples is omitted due to high economic costs. |  | Subset Size
---|---|---
Base Model | Algorithm | $k=100$ | $k=200$ | $k=400$ | $k=1000$
GPT-3.5-turbo | Random | 7.5$\pm$2.5% | 20.6$\pm$2.1% | 45.8$\pm$3.8% | –
MDFT-l | 8.3$\pm$1.5% | 19.3$\pm$2.7% | 42.2$\pm$2.1% | –
MDFT-g | 11.1$\pm$1.6% | 25.0$\pm$1.2% | 51.8$\pm$1.2% | –
Llama-2-7b | Random | 0.4$\pm$0.1% | 1.9$\pm$0.3% | 11.9$\pm$1.3% | 53.0$\pm$2.1%
MDFT-l | 0.8$\pm$0.3% | 2.0$\pm$0.4% | 12.4$\pm$3.0% | 54.1$\pm$2.4%
MDFT-g | 0.4$\pm$0.2% | 4.7$\pm$0.9% | 15.5$\pm$1.5% | 56.1$\pm$1.8%
Llama-3-8b | Random | 2.8$\pm$1.1% | 10.3$\pm$0.7% | 44.4$\pm$1.5% | 72.7$\pm$3.1%
MDFT-l | 3.9$\pm$1.0% | 8.8$\pm$1.5% | 46.8$\pm$3.8% | 73.2$\pm$1.8%
MDFT-g | 5.3$\pm$1.0% | 12.9$\pm$1.6% | 49.8$\pm$2.6% | 75.3$\pm$0.8%

Finally, we compare the performance between MDFT-l and Random in detail.
Figure 5 presents the win-tie-loss comparison between MDFT-l and Random.
MDFT-g is omitted from this comparison as it is consistently superior. In
Figure 5, MDFT-l wins most comparisons in both Blocksworld and Logistics,
further confirming the effectiveness of the maximum diversity fine-tuning
strategy.

Figure 5: Performance comparison between MDFT-l and Random.

##  6 Limitations

While the proposed algorithm enhances LLMs’ performance in automated planning
with higher sample efficiency, the planning capabilities acquired in one
domain are not transferable to new domains. Although the mixed-data fine-
tuning approach proves effective, it also introduces higher variance in
outcomes, indicating the need for further refinement and stability
enhancements. Addressing these limitations will be the focus of our future
work. Cross-domain transferability in planning remains an underexplored area,
highlighting a significant opportunity for further investigation.

##  7 Conclusion

In this paper, we demonstrated that fine-tuning significantly enhances the
planning capabilities of LLMs, even with limited data. To improve sample
efficiency in fine-tuning, we introduced the Maximum Diversity Fine-Tuning
(MDFT) strategy. Our clustering-based method can effectively solve the maximum
diversity problem, ensuring a broad and representative sample set. Our
proposed methods, MDFT-g, which uses graph embeddings, and MDFT-l, which uses
language embeddings, were shown to outperform random sampling. These findings
underscore the value of strategic sample selection in maximizing the
performance and efficiency of LLM fine-tuning for automated planning.

## References

  * Achiam et al. [2023] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_ , 2023.
  * Brown et al. [2020] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Cer et al. [2018] D. Cer, Y. Yang, S.-y. Kong, N. Hua, N. Limtiaco, R. S. John, N. Constant, M. Guajardo-Cespedes, S. Yuan, C. Tar, et al.  Universal sentence encoder.  _arXiv preprint arXiv:1803.11175_ , 2018.
  * Chen et al. [2023] B. Chen, C. Shu, E. Shareghi, N. Collier, K. Narasimhan, and S. Yao.  Fireact: Toward language agent fine-tuning.  _arXiv preprint arXiv:2310.05915_ , 2023.
  * Ding et al. [2023] N. Ding, Y. Qin, G. Yang, F. Wei, Z. Yang, Y. Su, S. Hu, Y. Chen, C.-M. Chan, W. Chen, et al.  Parameter-efficient fine-tuning of large-scale pre-trained language models.  _Nature Machine Intelligence_ , 5(3):220–235, 2023.
  * Ghosh [1996] J. B. Ghosh.  Computational aspects of the maximum diversity problem.  _Operations research letters_ , 19(4):175–181, 1996.
  * Guan et al. [2023] L. Guan, K. Valmeekam, S. Sreedharan, and S. Kambhampati.  Leveraging pre-trained large language models to construct and utilize world models for model-based task planning.  _Advances in Neural Information Processing Systems_ , 36:79081–79094, 2023.
  * Helmert [2006] M. Helmert.  The fast downward planning system.  _Journal of Artificial Intelligence Research_ , 26:191–246, 2006.
  * Howey et al. [2004] R. Howey, D. Long, and M. Fox.  Val: Automatic plan validation, continuous effects and mixed initiative planning using pddl.  In _16th IEEE International Conference on Tools with Artificial Intelligence_ , pages 294–301. IEEE, 2004.
  * Huang et al. [2022] W. Huang, P. Abbeel, D. Pathak, and I. Mordatch.  Language models as zero-shot planners: Extracting actionable knowledge for embodied agents.  In _International Conference on Machine Learning_ , pages 9118–9147. PMLR, 2022.
  * Liu et al. [2023] B. Liu, Y. Jiang, X. Zhang, Q. Liu, S. Zhang, J. Biswas, and P. Stone.  Llm+ p: Empowering large language models with optimal planning proficiency.  _arXiv preprint arXiv:2304.11477_ , 2023.
  * Long and Fox [2003] D. Long and M. Fox.  The 3rd international planning competition: Results and analysis.  _Journal of Artificial Intelligence Research_ , 20:1–59, 2003.
  * Martí et al. [2013] R. Martí, M. Gallego, A. Duarte, and E. G. Pardo.  Heuristics and metaheuristics for the maximum diversity problem.  _Journal of Heuristics_ , 19:591–615, 2013.
  * Pallagani et al. [2023] V. Pallagani, B. Muppasani, K. Murugesan, F. Rossi, B. Srivastava, L. Horesh, F. Fabiano, and A. Loreggia.  Understanding the capabilities of large language models for automated planning.  _arXiv preprint arXiv:2305.16151_ , 2023.
  * Rivlin et al. [2020] O. Rivlin, T. Hazan, and E. Karpas.  Generalized planning with deep reinforcement learning.  _arXiv preprint arXiv:2005.02305_ , 2020.
  * Silver et al. [2021] T. Silver, R. Chitnis, A. Curtis, J. B. Tenenbaum, T. Lozano-Pérez, and L. P. Kaelbling.  Planning with learned object importance in large problem instances using graph neural networks.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 35, pages 11962–11971, 2021.
  * Tinn et al. [2023] R. Tinn, H. Cheng, Y. Gu, N. Usuyama, X. Liu, T. Naumann, J. Gao, and H. Poon.  Fine-tuning large neural language models for biomedical natural language processing.  _Patterns_ , 4(4), 2023.
  * Touvron et al. [2023] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023.
  * Valmeekam et al. [2024] K. Valmeekam, M. Marquez, S. Sreedharan, and S. Kambhampati.  On the planning abilities of large language models-a critical investigation.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Van der Maaten and Hinton [2008] L. Van der Maaten and G. Hinton.  Visualizing data using t-sne.  _Journal of machine learning research_ , 9(11), 2008.
  * Wang et al. [2024] Z. Wang, S. Cai, G. Chen, A. Liu, X. S. Ma, and Y. Liang.  Describe, explain, plan and select: interactive planning with llms enables open-world multi-task agents.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Wei et al. [2022] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in neural information processing systems_ , 35:24824–24837, 2022.
  * Yao et al. [2022] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao.  React: Synergizing reasoning and acting in language models.  _arXiv preprint arXiv:2210.03629_ , 2022.
  * Yao et al. [2024] S. Yao, D. Yu, J. Zhao, I. Shafran, T. Griffiths, Y. Cao, and K. Narasimhan.  Tree of thoughts: Deliberate problem solving with large language models.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Yuan et al. [2023] H. Yuan, C. Zhang, H. Wang, F. Xie, P. Cai, H. Dong, and Z. Lu.  Skill reinforcement learning and planning for open-world long-horizon tasks.  _arXiv preprint arXiv:2303.16563_ , 2023.
  * Zhao et al. [2024] H. Zhao, M. Andriushchenko, F. Croce, and N. Flammarion.  Long is more for alignment: A simple but tough-to-beat baseline for instruction fine-tuning.  _arXiv preprint arXiv:2402.04833_ , 2024.
  * Zhou et al. [2024] C. Zhou, P. Liu, P. Xu, S. Iyer, J. Sun, Y. Mao, X. Ma, A. Efrat, P. Yu, L. Yu, et al.  Lima: Less is more for alignment.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Zhu et al. [2018] Y. Zhu, S. Lu, L. Zheng, J. Guo, W. Zhang, J. Wang, and Y. Yu.  Texygen: A benchmarking platform for text generation models.  In _The 41st international ACM SIGIR conference on research & development in information retrieval_, pages 1097–1100, 2018.


# PhasePerturbation: Speech Data Augmentation via Phase Perturbation for
Automatic Speech Recognition

Chengxi Lei<EMAIL_ADDRESS>School of Artificial Intelligence,
Dalian Maritime University,China ,  Satwinder Singh<EMAIL_ADDRESS>School of Mathematical and Computational Sciences,
Massey University,New Zealand ,  Feng Hou<EMAIL_ADDRESS>School of
Mathematical and Computational Sciences,
Massey University,New Zealand ,  Xiaoyun Jia<EMAIL_ADDRESS>Institute of Governance & School of Political Science and Public
Administration,
Shandong University,China and  Ruili Wang<EMAIL_ADDRESS>School of
Artificial Intelligence,
Dalian Maritime University,China

(2023)

###### Abstract.

Most of the current speech data augmentation methods operate on either the raw
waveform or the amplitude spectrum of speech. In this paper, we propose a
novel speech data augmentation method called PhasePerturbation that operates
dynamically on the phase spectrum of speech. Instead of statically rotating a
phase by a constant degree, PhasePerturbation utilizes three dynamic phase
spectrum operations, i.e., a randomization operation, a frequency masking
operation, and a temporal masking operation, to enhance the diversity of
speech data. We conduct experiments on wav2vec2.0 pre-trained ASR models by
fine-tuning them with the PhasePerturbation augmented TIMIT corpus. The
experimental results demonstrate 10.9% relative reduction in the word error
rate (WER) compared with the baseline model fine-tuned without any
augmentation operation. Furthermore, the proposed method achieves additional
improvements (12.9% and 15.9%) in WER by complementing the Vocal Tract Length
Perturbation (VTLP) and the SpecAug, which are both amplitude spectrum-based
augmentation methods. The results highlight the capability of
PhasePerturbation to improve the current amplitude spectrum-based augmentation
methods.

speech recognition, data augmentation, phase spectrum augmentation

††journalyear: 2023††copyright: acmlicensed††conference: ACM Multimedia Asia
Workshops; December 6–8, 2023; Tainan, Taiwan††booktitle: ACM Multimedia Asia
Workshops (MMAsia ’23 Workshops), December 6–8, 2023, Tainan, Taiwan††price:
15.00††doi: 10.1145/3611380.3628555††isbn: 979-8-4007-0326-3/23/12

##  1\. Introduction

Deep learning has demonstrated remarkable effectiveness in the field of
Automatic Speech Recognition (ASR) (Hinton et al., 2012). Extensive research
efforts have been devoted to improving ASR model network architectures,
leading to significant progress (Dahl et al., 2012; Graves et al., 2013; Hsu
et al., 2020; Winata et al., 2020; Singh et al., 2022). These ASR models often
require extensive volumes of training data. However, over 80% of languages are
categorized as low-resource languages, lacking such an amount of labeled data
(Singh et al., 2023b; Chiu et al., 2018). To address this issue, data
augmentation methods are used to generate synthetic speech data. There are
mainly two approaches for data augmentation for ASR: (i) Manipulating raw
waveforms to synthesize new data, such as introducing noise into clean audio
to create noisy samples (Hannun et al., 2014; Kim et al., 2017; Sainath et
al., 2016); (ii) Applying data augmentation directly to Mel-spectrum, which is
an amplitude spectrum. These Mel-spectrum based methods treat the amplitude
spectrum as an image and use the corresponding data augmentation approaches
(Ko et al., 2015; Jaitly and Hinton, 2013; Chan et al., 2019; Wang et al.,
2021; Song et al., 2020; Kim et al., 2021; Meng et al., 2021; Zhang et al.,
2018; Yun et al., 2019; Wang et al., 2022).

Lately, there has been a growing interest in utilizing phase information to
improve Generative Adversarial Network (GAN)-based vocoders used in speech
synthesis. Morrison et al. (Morrison et al., 2022) have shown that vocoder
generators could reliably rebuild signal phases from the Mel-spectrum, even
without the initial phase. PhaseAug (Lee et al., 2023) was proposed to address
the overfitting of the discriminator. The proposed approach rotated the phase
in the frequency domain to simulate one-to-many mapping.

Intuitively, it is valuable to investigate speech data augmentation techniques
applied to the phase spectrum of speech data, given the recent success in
effectively utilizing phase information in voc-oders. Thus, in this paper, we
propose a novel speech data augmentation method (PhasePerturbation) that
employs three augmentation operations on the phase spectrum: frequency
masking, temporal masking, and phase randomization. These operations draw
inspiration from SpecAug (Chan et al., 2019). We fine-tune two wav2vec2.0 pre-
trained ASR models (BASE LS-960 and LARGE LV-60K) with the augmented TIMIT
corpus, using different data augmentation methods. Our experimental results
reveal a remarkable 10.9% relative reduction in the word error rate (WER)
compared with the baseline model fine-tuned without any augmentation
operation. Additionally, the proposed method achieves further improvements
when combined with the VTLP and the SpecAug over the performance of the
baselines. Notably, combining our method with amplitude spectrum-based
augmentations leads to even lower WER, highlighting its potential for
enhancing existing amplitude-based augmentation methods.

Figure 1. Waveforms (top), Amplitude spectrums (middle), and Phase spectrums
(bottom) of a speaker repeating the same word twice.

##  2\. Background

###  2.1. Fourier Transform

The analog speech signal is turned into a real-value discrete time sequence
s[n] after sampling. Shifting the window function along the time axis to
divide $s[n]$ into time bins, these bins are sufficiently short to be treated
as stationary signals, where their characteristic properties remain constant
within each short time bin. The process of applying the Fourier transform to
each time bin is referred to as the Short-Time Fourier Transform (STFT). STFT
$S[k,m]=F(s(n))$ and its inverse transformation, the inversed short-time
Fourier transform (iSTFT) $s(n)=F^{-1}(S[k,m])$ are represented as equation 1
and 2:

(1) $S[k,m]=F(s(n))=\int_{-\infty}^{+\infty}s[n]w[n-md]e^{-j2\pi k/N}$

(2)
$x[n]=F^{-1}(S[k,m])=\frac{1}{C}\sum_{k=0}^{N-1}\sum_{m=-\infty}^{\infty}S[k,m]e^{j2\pi
k/N}$

where $N$ is the length of STFT, and $d$ is the hop length. $k$ and $m$ are
the index of the frequency bin and time bin of the time-frequency spectrum
respectively. The Hanning window is represented as $w[n]=1-\cos(2\pi n/N)/2$.
$C[n]=\sum_{m=-\infty}^{\infty}w[n-md]$ is the sum, at the time $n$, of all
the values of the window function moving along the time axis.

###  2.2. Phase Spectrum

The phase spectrum of a speech signal captures the phase angles associated
with different frequency components. Similar to the amplitude spectrum, the
phase spectrum can also be obtained through the Fourier Transform.
Mathematically, the phase spectrum $\phi[k,m]$ is given by:

(3) $\phi[k,m]=\arctan\frac{img(S[k,m])}{real(S[k,m])}$

Where $\phi[k,m]$ is the phase angle of the complex representation $S[k,m]$ at
frequency $k$. The phase spectrum $\phi[k,m]$ provides information about how
the different frequency components are aligned in time. In the phase spectrum,
each frequency component is associated with a phase angle that indicates the
timing relationship of that frequency’s oscillations.

Acoustic features like MFCC and filter bank output, extracted from the
amplitude spectrum, were previously preferred in speech recognition due to
their connection to perceived loudness and spectral characteristics. However,
their widespread use is a practical compromise driven by computational
constraints. Considering the limited resources, selecting a 39-dimensional
vector per frame for MFCC and an 80-dimensional vector per frame for filter
bank outputs appears computationally efficient, as opposed to directly
processing the raw 16 kHz speech signal, which results in a 400-sample frame
vector. Yet, with the continuous improvement in computer hardware performance
and the growing prevalence of pre-trained models in speech recognition (Singh
et al., 2023a; Chen and Rudnicky, 2023), recent research has increasingly
embraced the use of 16 kHz raw speech signals, incorporating phase information
to attain higher accuracy.

The phase spectrum significantly affects signal timbre, transients, and
speaker distinctions. As illustrated in Figure 1, raw signal waveforms may
vary, even when a speaker repeats the same sentence multiple times.

##  3\. Proposed Method

###  3.1. Augmentation Policy

To introduce more diversity to the phase information, we design randomization
augmentation operations for the phase spectrum. In addition, we adopt two
augmentation operations on the phase spectrum (i.e., frequency masking and
temporal masking) to increase the robustness of the partial loss of phase
information and the partial phase loss of small speech segments. The details
about the three augmentation operations are as follows.

####  3.1.1. Phase randomization

According to the equation 1, the value of the time-frequency spectrum is a
complex number that has amplitude and phase. Most previous data augmentation
methods ignore the phase information of this complex number, such as MFCC and
filter bank outputs. However, speech phase information contributes to the
diversity of the raw speech signal in the time domain. The raw signal of a
speaker repeating the same sentence several times can be of large difference
because the phase of the speech signal is highly random. Inspired by the
additive Gaussian white noise augmentation operation (Kharitonov et al.,
2021), we propose phase randomization to increase the diversity of speech
signals.

Initially, for phase randomization, we can assign independent random numbers
to each element in the two-dimensional phase spectrum matrix of the speech
signal, with dimensions $(x,y)$, where $x$ represents the total number of
frequency bins, and $y$ represents the total number of time bins. This
involves generating $x\times y$ independent random numbers from a standard
Normal distribution. These random numbers are then used to create a
randomization matrix through element-wise multiplication with the
corresponding phase spectrum matrix elements. Subsequently, the Inverse Short-
Time Fourier Transform (ISTFT) can be applied to obtain the phase-randomized
speech signal. However, this process introduces significant time-domain
distortion, adversely affecting speech intelligibility. Karras et al. (Karras
et al., 2020) attributed this distortion to substantial differences between
adjacent frequency bins in each time frame.

To address this perceptible distortion, instead of assigning random numbers to
each matrix element, we generate independent random numbers for each time bin,
which are shared among elements within a column. This method requires y
independent random numbers. The phase randomization operation can be
summarized as Algorithm 1. As demonstrated in Figure 2, our randomization
method yields diverse phase-randomized speech signals while preserving nearly
the same level of intelligibility as the original speech signal.

Input: The original speech signal $s[n]$

Output: The phase-randomized speech signal $S_{rand}[n]$

1 Apply STFT on $s[n]:S[k,m]=F(s(n))$

2 Calculate the phase spectrum matrix: $\phi[k,m]=argS([k,m])$

3 //$y$ is the column number of $\phi[k,m]$

4 while  _$i\leq y$ , _ do

5 Sample $\mu\sim N(1,\delta^{2})$

6 Multiplying elements in the $i$ th th column of $\phi[k,m]$ by $\mu$ times

7

8 end while

return $\phi[k,m]$

Algorithm 1 Phase randomization Figure 2. The original speech waveform (top)
and the phase-randomized speech waveform (bottom)

####  3.1.2. Frequency masking.

Frequency masking is implemented on the phase spectrum by masking a
consecutive range of frequency bins $[f_{0},f_{0}+f]$, where $f$ is initially
sampled from a uniform distribution from $0$ to the maximum frequency mask
width $F$, $f_{0}$ is selected from the interval $[0,v-f]$. Here, $v$
represents the total number of frequency bins in the phase spectrum.

####  3.1.3. Temporal masking.

Temporal masking is implemented on the phase spectrum by masking a consecutive
range of temporal bins $[t_{0},t_{0}+t]$, where $t$ is initially sampled from
a uniform distribution from $0$ to the maximum temporal mask width $T$, and
$t_{0}$ is selected from the interval $[0,\tau-t]$. Here, $\tau$ represents
the total number of temporal bins in the phase spectrum. Similar to SpecAug
(Chan et al., 2019), there is an upper limit imposed on the time mask to
ensure it doesn’t extend beyond $p$ times of $\tau$.

The phase spectrum is normalized to have a mean value of zero, thus setting
the masked value to zero is effectively the same as setting it to the mean
value. Figure 3 shows examples of the frequency masking and temporal masking
applied to a raw speech signal.

Figure 3. From top to bottom, the figures depict the phase spectrum of the
original speech signal without augmentation, frequency masking, and temporal
masking applied.

##  4\. Experiments and Results

###  4.1. Implementation Details

One-sided, Hanning windowed STFT with a size of 1024 and a hop length of 256
is used in our experiment. Phase augmentation parameters are summarized in
Table 1. $F$ and $T$ represent the maximum widths of the frequency mask and
the temporal mask, respectively. We consider policies where multiple frequency
and time masks are applied. $m_{F}$ and $m_{T}$ denote the number of frequency
and time masks applied. The multiple masks may overlap. $p$ represents the
maximum ratio of the temporal mask width to the speech length.

Table 1. The masking parameters. $F$ | $m_{F}$ | $T$ | $m_{T}$ | $p$
---|---|---|---|---
10 | 2 | 45 | 2 | 0.1

###  4.2. Model

For our experiments, we adopt the self-supervised Wav2vec2.0 (Baevski et al.,
2020) as the backbone for the ASR model. The Wav2vec2.0 model was pre-trained
with large amount of unlabeled data in a self-supervised manner. We fine-tune
the Wav2vec2.0 model with the augmented TIMIT corpus, using our proposed data
augmentation policies. To verify the performance of our phase augmentation
method on models with different sizes of parameters, two variants of
wav2vec2.0 models are used: BASE LS-960 and LARGE LV-60K. BASE and LAR-GE
models differ in Transformer architecture: The BASE model consists of 95
million parameters, featuring 12 transformer blocks, a model dimension of 768,
an inner dimension (FFN) of 3072, and 8 attention heads. In contrast, the
LARGE model is composed of 315 million parameters, incorporating 24
transformer blocks, a model dimension of 1024, an inner dimension of 4096, and
16 attention heads. The BASE LS-960 variant was pre-trained on LibriSpeech
960h (Panayotov et al., 2015), while the LARGE LV-60K model underwent pre-
training on 60k hours of unlabeled speech data from LibriVox (Kahn et al.,
2020). To reduce overfitting, the time-step and channel masking are applied on
the outputs of the encoder of wav2vec2.0 model as implemented in the original
wav2vec2.0 paper (Baevski et al., 2020).

Both pre-trained models are fine-tuned for 100 epochs with Adam optimizer with
a learning rate of $3e^{-5}$. The TIMIT dataset consists of default TRAIN and
TEST sets. To fine-tune the pre-trained wav2vec 2.0 models, we use a 90% TRAIN
set to train the model and a 10% TRAIN set for validation.

When fine-tuning the wav2vec2.0 models, 7 augmentation methods are applied to
the raw waveform speech data: (1) the original TIMIT dataset without
augmentation, (2) PhaseAug (Lee et al., 2023), (3) Vocal Tract Length
Perturbation (VTLP) (Jaitly and Hinton, 2013), (4) proposed PhasePerturbation,
(5) the combination of PhasePerturbation and VTLP, (6) SpecAug (Chan et al.,
2019), and (7) the combination of PhasePerturbation and SpecAug. The VTLP and
the SpecAug are both applied on the amplitude spectrum and then use iSTFT to
transform them back to the time domain.

Table 2. WER (%) of BASE LS-960 and LARGE LV-60K with five augmentation methods on the TIMIT. Models | Policies | WER (%)
---|---|---
BASE LS-960 | None | 21.6
PhaseAug[1] | 21.2
VTLP[2] | 20.2
PhasePerturbation | 19.4
PhasePerturbation+VTLP | 19.1
SpecAug[3] | 18.9
PhasePerturbation+SpecAug | 18.5
LARGE LV-60K | None | 20.1
PhaseAug[1] | 19.7
VTLP[2] | 18.8
PhasePerturbation | 17.9
PhasePerturbation+VTLP | 17.5
SpecAug[3] | 17.4
PhasePerturbation+SpecAug | 16.9

###  4.3. Experimental Results

In Table 2, our phase augmentation technique shows excellent performance
across all models, yielding 19.4% and 17.9% WER for the BASE LS-960 model and
the LARGE LV-60K model, respectively. The relative gains our method offers
over the original TIMIT dataset, are about 10.1% and 10.9% across both the
Wave2vec2.0 models. Our method outperforms the static PhaseAug approach by
1.8% WER. Moreover, our technique surpasses VTLP’s performance on both the
BASE LS-960 model and the LARGE LV-60K model. It is worth highlighting that
our dynamic phase augmentation method attains even lower WER (17.5% and 16.9%)
when combined with the VTLP and the SpecAug, which are both amplitude spectrum
augmentation operations. The results demonstrate the effectiveness of our
method for complementing existing augmentation methods that operate on the
amplitude spectrum.

##  5\. Conclusion

In this paper, we propose a novel dynamic phase spectrum augmentation method
for speech data, PhasePerturbation. Our method boosts augmentation diversity
by employing three phase spectrum operations: phase randomization, frequency
masking, and temporal masking. The experimental results demonstrate the
efficacy of our phase-based augmentation method in improving speech
recognition performance, especially when dealing with limited datasets. To
validate our method, we compare it to VTLP and PhaseAug, revealing that our
dynamic phase augmentation outperforms static phase operations. Our
experiments also demonstrate the potential of combining amplitude spectrum-
based methods with our phase spectrum-based method to enhance data diversity.

It is noteworthy that our phase augmentation technique impacts only the phase
spectrum, not the amplitude spectrum, like the Mel-spectrum. This limitation
restricts its usage to ASR systems that take raw waveform speech data as
input. However, this distinction sets our method apart, as many existing
augmentation techniques primarily manipulate the amplitude spectrum, as seen
in methods such as SpecAug++ (Wang et al., 2021) or CyclicAugment (Wang et
al., 2022). In our future research, we plan to explore the integration of
phase augmentation operations with state-of-the-art amplitude spectrum
methods.

###### Acknowledgements.

This work is supported by the 2020 Catalyst: Strategic New Zealand - Singapore
Data Science Research Programme Fund by the Ministry of Business, Innovation
and Employment (MBIE), New Zealand.

## References

  * (1)
  * Baevski et al. (2020) Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. 2020.  wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations. In _NeurIPS_. 12449–12460.  https://doi.org/10.48550/arXiv.2006.11477
  * Chan et al. (2019) William Chan, Yu Zhang, Chung-Cheng Chiu, and Barret Zoph. 2019.  SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition. In _Interspeech 2019_. 2613–2617.  https://doi.org/10.48550/arXiv.1904.08779
  * Chen and Rudnicky (2023) Li-Wei Chen and Alexander Rudnicky. 2023.  Exploring Wav2vec 2.0 Fine Tuning for Improved Speech Emotion Recognition. In _2023 IEEE International Conference on Acoustics, Speech and Signal Processing_. 1–5.  https://doi.org/10.1109/ICASSP49357.2023.10095036
  * Chiu et al. (2018) Chung-Cheng Chiu, Tara Sainath, Yonghui Wu, Rohit Prabhavalkar, Patrick Nguyen, Zhifeng Chen, Anjuli Kannan, Ron J. Weiss, Kanishka Rao, Ekaterina Gonina, Navdeep Jaitly, Bo Li, Jan Chorowski, and Michiel Bacchiani. 2018.  State-of-the-Art Speech Recognition with Sequence-to-Sequence Models. In _2018 IEEE International Conference on Acoustics, Speech and Signal Processing_. 4774–4778.  https://doi.org/10.1109/ICASSP.2018.8462105
  * Dahl et al. (2012) George Dahl, Dong Yu, Li Deng, and Alex Acero. 2012.  Context-Dependent Pre-Trained Deep Neural Networks for Large-Vocabulary Speech Recognition.  _2012 IEEE International Conference on Acoustics, Speech and Signal Processing_ 20, 1 (Jan. 2012), 30–42.  https://doi.org/10.1109/TASL.2011.2134090
  * Graves et al. (2013) Alex Graves, Abdel-Rahman Mohamed, and Geoffrey Hinton. 2013.  Speech recognition with deep recurrent neural networks. In _2013 IEEE International Conference on Acoustics, Speech and Signal Processing_. 6645–6649.  https://doi.org/10.1109/ICASSP.2013.6638947
  * Hannun et al. (2014) Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, and Andrew Y. Ng. 2014.  Deep Speech: Scaling up end-to-end speech recognition.  https://doi.org/10.48550/arXiv.1412.5567
  * Hinton et al. (2012) Geoffrey Hinton, Li Deng, Dong Yu, George Dahl, Abdel-Rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara Sainath, and Brian Kingsbury. 2012.  Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups.  _IEEE Signal Processing Magazine_ 29, 6 (Nov. 2012), 82–97.  https://doi.org/10.1109/MSP.2012.2205597
  * Hsu et al. (2020) Jui-Yang Hsu, Yuan-Jui Chen, and Hung-yi Lee. 2020.  Meta Learning for End-To-End Low-Resource Speech Recognition. In _2020 IEEE International Conference on Acoustics, Speech and Signal Processing_. 7844–7848.  https://doi.org/10.1109/ICASSP40776.2020.9053112
  * Jaitly and Hinton (2013) Navdeep Jaitly and Geoffrey Hinton. 2013.  Vocal Tract Length Perturbation (VTLP) improves speech recognition. In _Proceedings of the 30 th Int. Conf. Mach. Learn. 90 (2013)_. 42–51.  https://doi.org/
  * Kahn et al. (2020) Jacob Kahn, Morgane Riviere, and Wang Zheng. 2020.  Libri-Light: A Benchmark for ASR with Limited or No Supervision. In _2020 IEEE International Conference on Acoustics, Speech and Signal Processing_. IEEE, 7669–7673.  https://doi.org/10.1109/ICASSP40776.2020.9052942
  * Karras et al. (2020) Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. 2020.  Training Generative Adversarial Networks with Limited Data. In _NeurIPS_. 12104–12114.
  * Kharitonov et al. (2021) Eugene Kharitonov, Morgane Riviere, Gabriel Synnaeve, Lior Wolf, Pierre-Emmanuel Mazare, Matthijs Douze, and Emmanuel Dupoux. 2021.  Data Augmenting Contrastive Learning of Speech Representations in the Time Domain. In _2021 IEEE Spoken Language Technology Workshop (SLT)_. 215–222.  https://doi.org/10.1109/SLT48900.2021.9383605
  * Kim et al. (2017) Chanwoo Kim, Ananya Misra, Kean Chin, Thad Hughes, Arun Narayanan, Tara Sainath, and Michiel Bacchiani. 2017.  Generation of Large-Scale Simulated Utterances in Virtual Rooms to Train Deep-Neural Networks for Far-Field Speech Recognition in Google Home. In _Interspeech 2017_. 379–383.  https://doi.org/10.21437/Interspeech.2017-1510
  * Kim et al. (2021) Gwantae Kim, David Han, and Hanseok Ko. 2021.  SpecMix : A Mixed Sample Data Augmentation method for Training withTime-Frequency Domain Features. In _Interspeech 2021_. 546–550.  https://doi.org/10.21437/Interspeech.2021-103 arXiv:2108.03020 [cs].
  * Ko et al. (2015) Tom Ko, Vijayaditya Peddinti, Daniel Povey, and Sanjeev Khudanpur. 2015.  Audio augmentation for speech recognition. In _Interspeech 2015_. 3586–3589.  https://doi.org/10.21437/Interspeech.2015-711
  * Lee et al. (2023) Junhyeok Lee, Seungu Han, Hyunjae Cho, and Wonbin Jung. 2023.  PhaseAug: A Differentiable Augmentation for Speech Synthesis to Simulate One-to-Many Mapping. In _2023 IEEE International Conference on Acoustics, Speech and Signal Processing_. 1–5.  https://doi.org/10.1109/ICASSP49357.2023.10096374
  * Meng et al. (2021) Linghui Meng, Jin Xu, Xu Tan, Jindong Wang, Tao Qin, and Bo Xu. 2021.  MixSpeech: Data Augmentation for Low-Resource Automatic Speech Recognition. In _2021 IEEE International Conference on Acoustics, Speech and Signal Processing_. 7008–7012.  https://doi.org/10.1109/ICASSP39728.2021.9414483
  * Morrison et al. (2022) Max Morrison, Rithesh Kumar, Kundan Kumar, Prem Seetharaman, Aaron Courville, and Yoshua Bengio. 2022.  Chunked Autoregressive GAN for Conditional Waveform Synthesis. In _ICLR_.  https://doi.org/10.48550/arXiv.2110.10139
  * Panayotov et al. (2015) Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015.  Librispeech: An ASR corpus based on public domain audio books. In _2015 IEEE International Conference on Acoustics, Speech and Signal Processing_. 5206–5210.  https://doi.org/10.1109/ICASSP.2015.7178964
  * Sainath et al. (2016) Tara Sainath, Ron Weiss, and Kevin Wilson. 2016.  Factored spatial and spectral multichannel raw waveform CLDNNs. In _2016 IEEE International Conference on Acoustics, Speech and Signal Processing_. 5075–5079.  https://doi.org/10.1109/ICASSP.2016.7472644
  * Singh et al. (2023a) Satwinder Singh, Feng Hou, and Ruili Wang. 2023a.  A Novel Self-training Approach for Low-resource Speech Recognition.  https://doi.org/doi.org/10.48550/arXiv.2308.05269
  * Singh et al. (2022) Satwinder Singh, Ruili Wang, and Feng Hou. 2022.  Improved Meta Learning for Low Resource Speech Recognition. In _2022 IEEE International Conference on Acoustics, Speech and Signal Processing_. 4798–4802.  https://doi.org/10.1109/ICASSP43922.2022.9746899
  * Singh et al. (2023b) Satwinder Singh, Ruili Wang, Feng Hou, and Zhizhong Ma. 2023b.  Enhancing End-to-End Automatic Speech Recognition for Low-resource Punjabi Language using Synthesized Datasets.  https://doi.org/SSRN4181844
  * Song et al. (2020) Xingchen Song, Zhiyong Wu, Yiheng Huang, Dan Su, and Helen Meng. 2020.  SpecSwap: A Simple Data Augmentation Method for End-to-End Speech Recognition. In _Interspeech 2020_. 581–585.  https://doi.org/10.21437/Interspeech.2020-2275
  * Wang et al. (2021) Helin Wang, Yuexian Zou, and Wenwu Wang. 2021.  SpecAugment++: A Hidden Space Data Augmentation Method for Acoustic Scene Classification. In _Interspeech 2021_. 2613–2617.  https://doi.org/10.48550/arXiv.2103.16858
  * Wang et al. (2022) Zhihan Wang, Feng Hou, Yuanhang Qiu, Zhizhong Ma, Satwinder Singh, and Ruili Wang. 2022.  CyclicAugment: Speech Data Random Augmentation with Cosine Annealing Scheduler for Auotmatic Speech Recognition. In _Interspeech 2022_. 3859–3863.  https://doi.org/10.21437/Interspeech.2022-526
  * Winata et al. (2020) Genta Indra Winata, Samuel Cahyawijaya, Zihan Liu, Zhaojiang Lin, Andrea Madotto, Peng Xu, and Pascale Fung. 2020.  Learning Fast Adaptation on Cross-Accented Speech Recognition. In _Interspeech 2020_. 1276–1280.  https://doi.org/10.21437/Interspeech.2020-711
  * Yun et al. (2019) Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Seong Joon Oh, Youngjoon Yoo, and Junsuk Choe. 2019.  CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features. In _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_. 6022–6031.  https://doi.org/10.1109/ICCV.2019.00612
  * Zhang et al. (2018) Hongyi Zhang, Moustapha Cisse, Yann Dauphin, and David Lopez-Paz. 2018.  Mixup: Beyond Empirical Risk Minimization. In _In Proceedings of the 6th International Conference on Learning Representations._ 6008–6012.  https://doi.org/10.48550/arXiv.1710.09412


11institutetext: 1Yonsei University, 2Korea University
11email: {seil, danny0103, timespt<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
#  WoLF: Wide-scope Large Language Model Framework for CXR Understanding

Seil Kang1 1Yonsei University, 2Korea University 11email: {seil, danny0103,
timespt<EMAIL_ADDRESS><EMAIL_ADDRESS>Donghyun Kim1 1Yonsei
University, 2Korea University 11email: {seil, danny0103, timespt,
<EMAIL_ADDRESS><EMAIL_ADDRESS>Junhyeok Kim1 1Yonsei
University, 2Korea University 11email: {seil, danny0103, timespt,
<EMAIL_ADDRESS><EMAIL_ADDRESS>Hyo Kyung Lee2 1Yonsei
University, 2Korea University 11email: {seil, danny0103, timespt,
<EMAIL_ADDRESS><EMAIL_ADDRESS>Seong Jae Hwang1 1Yonsei
University, 2Korea University 11email: {seil, danny0103, timespt,
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Significant methodological strides have been made toward Chest X-ray (CXR)
understanding via modern vision-language models (VLMs), demonstrating
impressive Visual Question Answering (VQA) and CXR report generation
abilities. However, existing CXR understanding frameworks still possess
several procedural caveats. $\mathbf{\mathfrak{(1)}}$ Previous methods solely
use CXR reports, which are insufficient for comprehensive Visual Question
Answering (VQA), especially when additional health-related data like
medication history and prior diagnoses are needed. $\mathbf{\mathfrak{(2)}}$
Previous methods use raw CXR reports, which are often arbitrarily structured.
While modern language models can understand various text formats,
restructuring reports for clearer, organized anatomy-based information could
enhance their usefulness. $\mathbf{\mathfrak{(3)}}$ Current evaluation methods
for CXR-VQA primarily emphasize linguistic correctness, lacking the capability
to offer nuanced assessments of the generated answers. In this work, to
address the aforementioned caveats, we introduce WoLF, a Wide-scope Large
Language Model Framework for CXR understanding. To resolve
$\mathbf{\mathfrak{(1)}}$, we capture multi-faceted records of patients, which
are utilized for accurate diagnoses in real-world clinical scenarios.
Specifically, we adopt the Electronic Health Records (EHR) to generate
instruction-following data suited for CXR understanding. Regarding
$\mathbf{\mathfrak{(2)}}$, we enhance report generation performance by
decoupling knowledge in CXR reports based on anatomical structure even within
the attention step via masked attention. To address $\mathbf{\mathfrak{(3)}}$,
we introduce an AI-evaluation protocol optimized for assessing the
capabilities of LLM. Through extensive experimental validations, WoLF
demonstrates superior performance over other models on MIMIC-CXR in the AI-
evaluation arena about VQA (up to $+9.47\%$p mean score) and by metrics about
report generation ($+7.3\%$p BLEU-1).

###### Keywords:

CXR Understanding LLM Framework Instruction Tuning

##  1 Introduction

Figure 1: Comparisons with other models for VQA scenario given a CXR image.
Green thumbs indicate the quality of the response is good (accurate, helpful),
while red thumbs indicate bad (inaccurate, evasive), with respect to target
answers.

Recent years have witnessed significant progress in the field of Chest X-ray
(CXR) understanding, particularly through downstream tasks like Visual
Question Answering (VQA) and automated report generation. Despite considerable
advancements, we raise issues that models engaged in Chest X-ray (CXR)
understanding persistently encounter several challenges from a framework
standpoint. $\mathbf{\mathfrak{(1)}}$ Existing approaches [13, 24]
predominantly depend on CXR reports for supervised learning, overlooking the
crucial aspect of incorporating patients’ personalized health records, which
are diagnoses-supportive in real-world clinical scenarios.
$\mathbf{\mathfrak{(2)}}$ Additionally, the performance of report generation
is constrained by the unstructured format of CXR reports. Unstructured raw CXR
reports, exemplified by Fig. 2(b), impede the ability of models to learn CXR
anatomical structures in supervised learning settings, owing to their non-
intuitive format. $\mathbf{\mathfrak{(3)}}$ Lastly, the existing evaluation
metrics for CXR-VQA primarily focus on the correctness of answers, which falls
short in assessing the generative language models’ comprehensive understanding
of CXR imagery.

To tackle the issues illustrated above, we introduce WoLF, a Wide-scope Large
Language Model Framework for CXR understanding. We will delve into the
specifics of our approach, detailing the innovative solutions we develop for
each challenge:

$\mathbf{\mathfrak{(1)}}$ For more in-depth use of such systems in practice,
as exemplified in Fig. 1, the model must consider various patient records,
including Electronic Health Records (EHR). Thus, we hypothesize that
incorporating patients’ personalized EHR records can enhance the CXR
understanding of vision-language models. To validate this hypothesis, we
introduce Health-specific Instruction Tuning (HIT) to deal with the existing
limitations that training merely relies on CXR reports.

$\mathbf{\mathfrak{(2)}}$ Unorganized CXR reports restrict the advancement in
report generation tasks. To push the envelope, we present Anatomy-Specific
Knowledge decoupling (ASK) to separate the reports into anatomy-specific
findings. The generated targets give a model a direct understanding of a
specific anatomical structure, without being disturbed by other structures.
Synchronized with ASK, we introduce Anatomy-localizing Masked Attention (AMA)
that promotes independent learning on each anatomical structure.

$\mathbf{\mathfrak{(3)}}$ Current evaluation methods for CXR-VQA mostly
emphasize linguistic correctness. These methods are incapable of assessing the
responses from generative language models across a wide range. Inspired by
[12, 19], we provide a novel AI-evaluation protocol that is well-suited to
generative language models across dimensions of Accuracy, Helpfulness,
Relevance, Hallucination, and Universality. Through our extensive AI
evaluation, we can discern the extent to which models understand CXR from
their VQA results, rather than just evaluating the correctness of the models’
responses.

To sum up, the contribution of our model can be described at the macro and
micro level, respectively; Macroscopically, our framework covers data
reformulation, training method to improve CXR understanding, and AI-evaluation
protocol. Microscopically, (i) we present a novel instruction-following data
tuning method called Health-specific Instruction Tuning (HIT) designed for
interplay between personalized health records and visual representations of
CXR. (ii) We propose Anatomy-Specific Knowledge decoupling (ASK), for
hierarchically breaking down a radiology report by anatomical structures.
Furthermore, we present Anatomical-localizing Masked Attention to support the
merits of decoupled data from ASK, enabling expertised visual-language
comprehension for each anatomical structure. (iii) As the final step of the
framework, we introduce AI-evaluation for advanced analysis of our model. This
evaluates the broad capabilities of generative language models on the VQA
task. (iv) Through these methods, our study achieved state-of-the-art
performance in the report generation and VQA tasks on MIMIC-CXR [10] and IU-
Xray [7].

Figure 2: Data generation overview of HIT and ASK: (a) We generate health-
specific instruction-following dataset. In (a), Cyan and orange sequences are
queries about EHR and findings in CXR respectively. (b) We reorganize original
CXR reports into sequences of anatomy-specific structures through the use of a
knowledge graph, $G$.

##  2 WoLF: Wide-scope Large Language Model Framework

We introduce Wide-scope Large Language Model Framework (WoLF). WoLF
establishes its framework through the following macro-level steps:

$\mathop{\text{Data
Reformulation}}_{\text{\normalsize(Sec.~{}\ref{sec:2-1})}}\rightarrow\mathop{\text{Model
Training}}_{\text{\normalsize(Sec.~{}\ref{sec:2-2})}}\rightarrow\mathop{\text{AI-
evaluation}}_{\text{\normalsize(Sec.~{}\ref{sec:2-3})}}$

In Sec. 2.1, we describe our innovative data reformulation scheme designed for
VQA and report generation, the two main CXR understanding tasks. In Sec. 2.2,
we introduce a two-stage training approach to further enhance the model’s
performance in these tasks. Finally, in Sec. 4, we propose an AI-evaluation
protocol tailored for assessing generative language models in the CXR-VQA.

###  2.1 Data Reformulation

To train WoLF as a framework that excels in performing VQA and report
generation tasks for CXR, we propose two data reformulation methods: Health-
specific Instruction Tuning (HIT) for VQA, and Anatomy-Specific Knowledge
decoupling (ASK) for report generation.

Algorithm 1 HIT: Health-specific Instruction Tuning

Input: EHR QA-set: {$q^{\text{ehr}}$, $a^{\text{ehr}}$}, VQA QA-set:
{$q^{\text{vqa}}$, $a^{\text{vqa}}$}, # of {Patients, Studies}: {$P$,
$N_{p}$}, Open-source Language Model: $f_{\phi}$

$\text{global }S\leftarrow\texttt{SystemPrompt}$

for $p=1$ to $P$ do

for $i=1$ to $N_{p}$ do

$Q\leftarrow(q_{pi}^{\text{ehr}},q_{pi}^{\text{vqa}})$

$A\leftarrow(a_{pi}^{\text{ehr}},a_{pi}^{\text{vqa}})$

${S}_{i}\leftarrow$PromptGenerator($Q,A,f_{\phi}$)

${S}\leftarrow\text{concatenate}({S},{{S}_{i}})$

Output: Generated ${S}$

function PromptGenerator($Q,A,f$)

for $q_{t},a_{t}$ in $\\{{Q},{A}\\}$ do

${S}\leftarrow\text{concatenate}({S},{q_{t}},{a_{t}})$

return $f({S})$

Algorithm 2 ASK: Anatomy-specific Knowledge decoupling

Input: Knowledge graph: $G$, Anatomical structures: $O$, # of {Patients,
Studies, Anatomical structures}: {$P$, $N_{p}$, $M$}, CXR Reports: $R$

${D^{*}}\leftarrow\emptyset$

Def. 1: $R=\\{r_{i}|{i=1,2,\dots,N_{p}}\\}$

for $p=1$ to $P$ do

for $i=1$ to $N_{p}$ do

$\text{Append }\textsc{Decoupler}(O,G,r_{i})\text{ to }{D^{*}}$

Output: Decoupled $D^{*}$

function Decoupler($O,G,r$)

Def. 2: $O=\\{o_{m}|{m=1,2,\dots,M}\\}$

${D}\leftarrow\texttt{EmptyString("")}$

${T}\leftarrow\texttt{AnatomyTag}$

for $m=1$ to $M$ do

${D}\leftarrow\text{concatenate}({D},{{T}},{G(r,o_{m})})$

return ${D}$

HIT. LLaVA [18] and LLaVA-Med [14] have shown the effectiveness of explicitly
tuned data combined with vision-language instruction tuning, enabling thorough
reasoning about vision-language data. Specifically, to encode images into
their visual features, GPT-4 [21] is used as a text-only input teacher. The
newly generated data from the teacher takes the form of a multi-turn
conversation between the USER and the ASSISTANT. Inspired by LLaVA [18], our
approach, HIT (Alg. 1), focuses on CXR understanding, in particular, CXR VQA.
That is, we generate a novel health-specific instruction-following dataset of
multi-turn conversations that comprise EHR and the CXR findings. For instance,

  * User: Has patient been diagnosed with contact or exposure to tobacco smoke?

  * Assistant: Yes, the patient has been exposured with smoking environment.

  * User: <IMG> Is there any sign of the pneumonia in the right apical zone?

  * Assistant: Yes, the image has evidence of pneumonia in right apical zone.

<IMG> is a placeholder for image embeddings. HIT employs dataset [2]
constructed from MIMIC-IV [9] and MIMIC-CXR [10].

ASK. During training, models struggle to recognize individual anatomical
structures in unstructured CXR reports. For better report generation, we need
CXR reports organized by anatomy in supervised learning. To this end, we
present ASK to reorganize the CXR reports for more accurate report generation.
As explicated in Zhang et al. [30], attributes in the CXR reports are
classified based on the knowledge graph $G$. We separate sentences of CXR
reports based on anatomy-specific diseases (e.g., {pneumonia,
edema}$\rightarrow$lung). Eventually, we obtain target data consisting of
AnatomyTag and its findings (e.g., ### Heart: Moderate cardiomegaly is
present.). In the prior study, ITA [25] trains decoder heads to generate
sentences for anatomical structures and aggregates training losses. In
contrast, we directly dissociate the CXR report into independent anatomical
sentences (Alg. 2). Next, we describe how we use this refined CXR instruction
data to train WoLF.

###  2.2 Model Training

Figure 3: Overview of the training phase. (a) The input embedding ${{\bf
H}}_{(\cdot)}$ consists of visual embedding from the adapter and language
embeddings from $t$-turn conversations generated by HIT. Cyan and orange
sequences are queries about EHR and findings in CXR respectively. (b) For an
anatomical structure $o_{m}$, its sequence embedding is denoted by ${{\bf
H}}^{o_{m}}$. Organized CXR reports by ASK are utilized as input for training.

The training phase of WoLF is divided into two stages (Fig. 3). In stage 1,
the model is trained specifically for the VQA task, while stage 2 focuses on
the report generation task, leveraging the CXR understanding trained in the
previous stage. Thus, WoLF holistically allows both VQA and report generation
based on its superior CXR understanding.

Stage 1. The proposed dataset comprises the following components at the token
level; images ${{\bf X}}_{\text{V}}$, system prompt ${{\bf X}}_{\text{sys}}$,
user questions ${{\bf X}}_{\text{q}}$, and assistant answers ${{\bf
X}}_{\text{a}}$. Precisely, for a sequence of length $L$, we compute the
probability of the assistant answers ${{\bf X}}_{\text{a}}$ as target answers
by

$\small p({{\bf X}}_{\text{a}}|{{\bf X}}_{\text{V}},{{\bf
X}}_{\text{sys}},{{\bf
X}}_{\text{q}})=\prod_{i=1}^{L}p_{\theta}(x_{\text{a},i}|{{\bf
X}}_{\text{V}},{{\bf X}}_{\text{sys}},{{\bf X}}_{\text{q},<i},{{\bf
X}}_{\text{a},<i}),$ (1)

where $\theta$ represents trainable parameters. This outlines the objective of
the WoLF training phase through HIT instructions. The model seeks to optimize
the probability $p_{\theta}$ of predicting the current token $x_{\text{a},i}$,
conditioned on ${{\bf X}}_{\text{V}},{{\bf X}}_{\text{sys}},{{\bf
X}}_{\text{q}}$, and ${{\bf X}}_{\text{a},<i}$ which denotes the preceding
answer tokens before $x_{\text{a},i}$. As shown in Fig. 2(a), adapter and LLM
are trainable but the image encoder is non-trainable. Stage 2. To effectively
utilize decoupled CXR reports from ASK, we present Anatomy-localizing Masked
Attention for the second training phase. We define $o_{m}$ to be a specific
anatomical structure of $m^{\text{th}}$. As shown in Fig. 3(b), when the model
predicts ${{\bf X}}^{o_{m}}$, it attends exclusively on ${{\bf
H}}^{o_{m}}_{\text{q}},{{\bf X}}_{\text{V}},\text{and }{{\bf
X}}_{\text{sys}}$. The prediction process of our model can be described in an
auto-regressive manner, as follows:

$\small p({{\bf X}}^{o_{m}}|{{\bf X}}_{\text{V}},{{\bf
X}}_{\text{sys}})=\prod_{i=1}^{L^{o_{m}}}p_{\theta}(x^{o_{m}}_{i}|{{\bf
X}}_{\text{V}},{{\bf X}}_{\text{sys}},{{\bf X}}^{o_{m}}_{<i}),$ (2)

which describes the objective of a model during the second stage for a
sequence of ${o_{m}}$. The model predicts $L^{o_{m}}$ tokens for the sequence
of $o_{m}$ during training. As a result, we focus on guiding the model to
learn independent anatomical comprehension in a CXR. Note that we use the
final model for all task inferences.

###  2.3 AI-evaluation

Figure 4: Our AI-evaluation protocol overview. Green and red are positive and
negative feedback from the evaluator, respectively. Only our predictions
received positive feedback.

We now present a quantitative analysis of LLMs through AI-evaluation in CXRVQA
[2] exploiting prior studies [12, 19]. Traditional evaluations, particularly
in Visual Question Answering (VQA), have focused merely on correctness.
However, we measure five metrics (Accuracy, Helpfulness, Relevance,
Hallucination, Universality) within the external AI evaluator [1, 23]. As
shown in Fig. 4, both our model and another comparable model are evaluated by
mirroring the human cognitive process, employing a prompt detailed in Supp.
Fig. 6. Both models generate responses based on the input CXR and visual
question. The evaluator receives text-only inputs which are questions,
answers, responses from ours and a comparable model, and a prompt. The
judgment, in Fig. 4, is a human-like evaluation through obvious criteria. To
offset position bias, we average the rating for the evaluation set and that
for the position-swapped set to get the final rating.

Table 1: LLM capability comparison using relative scores on CXRVQA [2] test-
set. The scores (_mean_($\pm$_std_)) are converted to a 100-point scale.

Method |  Gemini [23] |
---|---|---
Accuracy$\uparrow$ | Helpfulness$\uparrow$ | Relevance$\uparrow$ | Hallucination$\uparrow$ | Universality$\uparrow$ | Mean$\uparrow$
LLaVA-Med [14] | 64.32($\pm$0.35) | 66.29($\pm$1.02) | 67.32($\pm$1.11) | 71.97($\pm$2.43) | 67.12($\pm$0.81) | 67.41($\pm$0.25)
XrayGPT [24] | 30.75($\pm$0.79) | 34.59($\pm$0.43) | 38.19($\pm$0.78) | 55.11($\pm$3.16) | 38.85($\pm$0.82) | 39.50($\pm$0.41)
LLM-CXR [13] | 61.20($\pm$0.97) | 69.67($\pm$0.32) | 69.29($\pm$1.09) | 77.88($\pm$0.73) | 67.56($\pm$0.29) | 69.12($\pm$0.37)
WoLF | 73.03($\pm$0.64) | 73.19($\pm$0.87) | 72.34($\pm$0.42) | 79.02($\pm$0.67) | 75.95($\pm$0.49) | 74.70($\pm$0.34)
|  PaLM-2 [1] |
| Accuracy$\uparrow$ | Helpfulness$\uparrow$ | Relevance$\uparrow$ | Hallucination$\uparrow$ | Universality$\uparrow$ | Mean$\uparrow$
LLaVA-Med [14] | 67.38($\pm$0.15) | 67.35($\pm$0.76) | 66.47($\pm$2.30) | 72.51($\pm$4.33) | 67.51($\pm$0.36) | 68.25($\pm$0.31)
XrayGPT [24] | 33.54($\pm$0.59) | 45.04($\pm$0.30) | 36.43($\pm$0.65) | 51.11($\pm$0.91) | 40.94($\pm$1.10) | 41.41($\pm$0.25)
LLM-CXR [13] | 68.73($\pm$0.19) | 70.03($\pm$1.39) | 69.91($\pm$1.95) | 74.80($\pm$3.53) | 69.80($\pm$0.07) | 70.66($\pm$0.06)
WoLF | 79.54($\pm$0.77) | 79.93($\pm$1.22) | 79.21($\pm$0.58) | 85.30($\pm$3.74) | 78.18($\pm$0.91) | 80.13($\pm$1.17)

##  3 Experiments

We show the VQA results using AI-evaluation and report generation performance
of WoLF and other existing methods. Further qualitative results, ablations,
and experiments on the same dataset [10] can be found in Supp. Fig. 7.

Training Details. Vicuna-7b [6] is used for core LLM with LoRA [8] fine-
tuning. CLIP-ViT-L-14 [22] is used as an image encoder. The batch size is set
to 64. The learning rate is set to $2\text{e-}5$. Each stage is trained for 2
epochs.

VQA performance based on AI-evaluation. We use CXRVQA [2] test dataset to show
AI-evaluation results. The test dataset comprises 11,309 VQAs for CXR mainly
focusing on findings and visual questions that require interpreting images. We
analyze with comparable existing models of which the source code is publicly
available and reproducible. We ensure adherence to domain consistency since
the CXRVQA is derived from the MIMIC-CXR [10]. In Table 1, we evaluate scores
for each component (Accuracy, Helpfulness, Relevance, Hallucination,
Universality) to Gemini [23] and PaLM-2 [1] by querying them 4-times each
while swapping answer positions (temperature $\tau=1.0$). Moreover, inspired
by RLAIF [12], we instruct the AI evaluator to measure the win-rate using a
modified prompt referred to Supp. Fig. 6. Specifically, let a visual question
set be $\\{q_{i}|{i=1,2,\dots,n}\\}$. For each question $q_{i}$, we tell the
AI evaluator to prioritize responses from evaluated models instead of scoring
them. We infer every pair of candidates to the AI evaluator swapping answer
position (temperature $\tau=0.01$). See Supp. Table 4(b) for the result of
win-rate.

Table 2: Report generation performance comparisons against other methods on
MIMIC-CXR [10] and IU-Xray [7]. For a fair comparison with other
methodologies, we directly quoted from the published literature.

Method | BLEU-1$\uparrow$ | BLEU-2$\uparrow$ | BLEU-3$\uparrow$ | BLEU-4$\uparrow$ | METEOR$\uparrow$ | ROUGE-L$\uparrow$
---|---|---|---|---|---|---
55pt. | MIMIC | IU | MIMIC | IU | MIMIC | IU | MIMIC | IU | MIMIC | IU | MIMIC | IU
R2Gen [5] | 0.353 | 0.470 | 0.218 | 0.304 | 0.145 | 0.219 | 0.103 | 0.165 | 0.142 | 0.187 | 0.277 | 0.371
R2GenCMN [4] | 0.353 | 0.475 | 0.218 | 0.309 | 0.148 | 0.222 | 0.106 | 0.170 | 0.142 | 0.191 | 0.278 | 0.375
M2Tr [20] | 0.378 | 0.486 | 0.232 | 0.317 | 0.154 | 0.232 | 0.107 | 0.173 | 0.145 | 0.192 | 0.272 | 0.390
CMCL [15] | 0.344 | 0.473 | 0.217 | 0.305 | 0.140 | 0.217 | 0.097 | 0.162 | 0.133 | 0.186 | 0.281 | 0.378
CMN [3] | 0.353 | 0.475 | 0.218 | 0.309 | 0.148 | 0.222 | 0.106 | 0.170 | 0.142 | 0.191 | 0.278 | 0.375
CA [17] | 0.350 | 0.492 | 0.219 | 0.314 | 0.152 | 0.222 | 0.109 | 0.169 | 0.151 | 0.193 | 0.283 | 0.380
PPKED [16] | 0.360 | 0.483 | 0.224 | 0.315 | 0.149 | 0.224 | 0.106 | 0.168 | 0.149 | 0.376 | 0.284 | 0.351
TransSQ [11] | 0.423 | 0.484 | 0.261 | 0.333 | 0.171 | 0.238 | 0.116 | 0.175 | 0.168 | 0.207 | 0.286 | 0.415
MCGN [27] | 0.373 | - | 0.235 | - | 0.162 | - | 0.120 | - | 0.143 | - | 0.282 | -
ITA [25] | 0.395 | 0.505 | 0.253 | 0.340 | 0.170 | 0.247 | 0.121 | 0.188 | 0.147 | 0.208 | 0.284 | 0.382
METransformer [26] | 0.386 | 0.483 | 0.250 | 0.322 | 0.169 | 0.228 | 0.124 | 0.172 | 0.152 | 0.192 | 0.291 | 0.380
WoLF | 0.496 | 0.517 | 0.335 | 0.366 | 0.233 | 0.261 | 0.165 | 0.199 | 0.187 | 0.412 | 0.370 | 0.456

Figure 5: Qualitative result of WoLF on report generation. Each highlighted
color is mapped to the semantics of specific findings.

Report Generation. As shown in Table 3, we evaluate the quantitative result of
automated CXR report generation. Our evaluation follows the official dataset
split of MIMIC-CXR [10] and the dataset split with a proportion as [5, 26] on
IU-Xray [7]. We measured performance through metrics such as BLEU, METEOR, and
ROUGE. WoLF outperformed the previous state-of-the-art method by a significant
margin. Furthermore, as demonstrated in Fig. 5, our qualitative results
validate that WoLF consistently generates content aligned with the Ground
Truth (GT) report for various anatomical structures. See orange box in Supp.
Fig. 7 for more qualitative report generation results. Ablation and Other
Studies. In Supp. Table 4(a), we conduct ablation studies of Table 1. We
examine our model learned without EHR and with EHR to the model during
inference. Our ablation study For an additional comparison, please see Supp.
Table 1(b), which shows traditional Visual Question Answering (VQA)
experiments using the ELIXR [28] framework on the MIMIC-CXR dataset. In these
experiments, we have achieved state-of-the-art results.

##  4 Conclusion and Future Work

This paper presents the first comprehensive exploration of LLM-based framework
for understanding CXRs, encompassing data reformulation, model training, and
evaluation strategies. We incorporate EHR into the model to enhance CXR
understanding and generate instruction-following data reflecting real-world
clinical processes. Furthermore, we refine CXR reports by decoupling them
based on anatomical structures for training and introduce a novel masked
attention mechanism to improve report generation performance. Additionally,
our innovative AI evaluation protocol enables the assessment of LLMs from
diverse perspectives. We anticipate that our contributions, coupled with real-
time EHR retrieval pipelines, will yield more adaptable frameworks for
clinical decision-making.

## References

  * [1] Anil, R., Dai, A.M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al.: Palm 2 technical report. arXiv preprint arXiv:2305.10403 (2023)
  * [2] Bae, S., Kyung, D., Ryu, J., Cho, E., Lee, G., Kweon, S., Oh, J., Ji, L., Eric, I., Chang, C., et al.: Ehrxqa: A multi-modal question answering dataset for electronic health records with chest x-ray images. In: Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2023)
  * [3] Chen, Z., Shen, Y., Song, Y., Wan, X.: Cross-modal memory networks for radiology report generation. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pp. 5904–5914 (2021)
  * [4] Chen, Z., Shen, Y., Song, Y., Wan, X.: Generating radiology reports via memory-driven transformer. In: Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Aug 2021)
  * [5] Chen, Z., Song, Y., Chang, T., Wan, X.: Generating radiology reports via memory-driven transformer. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020 (2020)
  * [6] Chiang, W.L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J.E., Stoica, I., Xing, E.P.: Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality (2023), https://lmsys.org/blog/2023-03-30-vicuna/
  * [7] Demner-Fushman, D., Kohli, M.D., Rosenman, M.B., Shooshan, S.E., Rodriguez, L., Antani, S., Thoma, G.R., McDonald, C.J.: Preparing a collection of radiology examinations for distribution and retrieval. Journal of the American Medical Informatics Association 23(2), 304–310 (2016)
  * [8] Hu, E.J., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al.: Lora: Low-rank adaptation of large language models. In: International Conference on Learning Representations (2021)
  * [9] Johnson, A.E., Bulgarelli, L., Shen, L., Gayles, A., Shammout, A., Horng, S., Pollard, T.J., Moody, B., Gow, B., Lehman, L.w.H., et al.: Mimic-iv, a freely accessible electronic health record dataset. Scientific data 10(1), 1 (2023)
  * [10] Johnson, A.E., Pollard, T.J., Berkowitz, S.J., Greenbaum, N.R., Lungren, M.P., Deng, C.y., Mark, R.G., Horng, S.: Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data 6(1), 317 (2019)
  * [11] Kong, M., Huang, Z., Kuang, K., Zhu, Q., Wu, F.: Transq: Transformer-based semantic query for medical report generation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 610–620 (2022)
  * [12] Lee, H., Phatale, S., Mansoor, H., Lu, K., Mesnard, T., Bishop, C., Carbune, V., Rastogi, A.: Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267 (2023)
  * [13] Lee, S., Kim, W.J., Ye, J.C.: Llm itself can read and generate cxr images. arXiv preprint arXiv:2305.11490 (2023)
  * [14] Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang, J., Naumann, T., Poon, H., Gao, J.: Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems 36 (2024)
  * [15] Liu, F., Ge, S., Wu, X.: Competence-based multimodal curriculum learning for medical report generation. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (2021)
  * [16] Liu, F., Wu, X., Ge, S., Fan, W., Zou, Y.: Exploring and distilling posterior and prior knowledge for radiology report generation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 13753–13762 (2021)
  * [17] Liu, F., Yin, C., Wu, X., Ge, S., Zhang, P., Sun, X.: Contrastive attention for automatic chest x-ray report generation. In: Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. pp. 269–280 (2021)
  * [18] Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. Advances in neural information processing systems 36 (2024)
  * [19] Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., Zhu, C.: G-eval: Nlg evaluation using gpt-4 with better human alignment. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. pp. 2511–2522 (2023)
  * [20] Nooralahzadeh, F., Gonzalez, N.P., Frauenfelder, T., Fujimoto, K., Krauthammer, M.: Progressive transformer-based generation of radiology reports. In: Findings of the Association for Computational Linguistics: EMNLP (2021)
  * [21] OpenAI: Gpt-4 technical report (2023)
  * [22] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763 (2021)
  * [23] Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A.M., Hauth, A., et al.: Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023)
  * [24] Thawkar, O., Shaker, A., Mullappilly, S.S., Cholakkal, H., Anwer, R.M., Khan, S., Laaksonen, J., Khan, F.S.: Xraygpt: Chest radiographs summarization using medical vision-language models. arXiv preprint arXiv:2306.07971 (2023)
  * [25] Wang, L., Ning, M., Lu, D., Wei, D., Zheng, Y., Chen, J.: An inclusive task-aware framework for radiology report generation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 568–577 (2022)
  * [26] Wang, Z., Liu, L., Wang, L., Zhou, L.: Metransformer: Radiology report generation by transformer with multiple learnable expert tokens. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11558–11567 (2023)
  * [27] Wang, Z., Tang, M., Wang, L., Li, X., Zhou, L.: A medical semantic-assisted transformer for radiographic report generation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention (2022)
  * [28] Xu, S., Yang, L., Kelly, C., Sieniek, M., Kohlberger, T., Ma, M., Weng, W.H., Kiraly, A., Kazemzadeh, S., Melamed, Z., et al.: Elixr: Towards a general purpose x-ray artificial intelligence system through alignment of large language models and radiology vision encoders. arXiv preprint arXiv:2308.01317 (2023)
  * [29] Yang, L., Wang, Z., Zhou, L.: Medxchat: Bridging cxr modalities with a unified multimodal large model. arXiv preprint arXiv:2312.02233 (2023)
  * [30] Zhang, Y., Wang, X., Xu, Z., Yu, Q., Yuille, A., Xu, D.: When radiology report generation meets knowledge graph (2020)

—– Supplementary Material —– WoLF: Wide-scope Large Language Model Framework
for Chest X-ray Understanding Seil Kang1, Donghyun Kim1, Junhyeok Kim1, Hyo
Kyung Lee2, Seong Jae Hwang1

Table 3: (a) Ablation study of main paper Table 1. - for a given set of WoLF
trained without EHRXQA [2]. + for a given set of WoLF given corresponding
patient EHR while inference. (b) Win-rate results for each models. For
instance, a $>$ 50% win-rate signifies left model beats right model in more
than a half of the questions.

Method |  Gemini [23] |
---|---|---
Accuracy$\uparrow$ | Helpfulness$\uparrow$ | Relevance$\uparrow$ | Hallucination$\uparrow$ | Universality$\uparrow$ | Mean$\uparrow$
WoLF- | 71.58($\pm$0.77) | 72.10($\pm$0.51) | 69.27($\pm$1.09) | 78.14($\pm$2.13) | 74.96($\pm$0.77) | 73.21($\pm$0.12)
WoLF |  73.03($\pm$0.64) |  73.19($\pm$0.87) |  72.34($\pm$0.42) |  79.02($\pm$0.67) |  75.95($\pm$0.49) |  74.70($\pm$0.34)
WoLF+ | 77.80($\pm$0.24) | 74.55($\pm$0.53) | 73.03($\pm$0.63) | 82.49($\pm$0.83) | 77.73($\pm$0.43) | 77.12($\pm$0.16)
|  PaLM-2 [1] |
| Accuracy | Helpfulness | Relevance | Hallucination | Universality | Mean
WoLF- | 77.51($\pm$0.98) | 76.77($\pm$1.36) | 74.42($\pm$2.75) | 81.93($\pm$1.19) | 77.45($\pm$2.60) | 77.61($\pm$1.26)
WoLF |  79.54($\pm$0.77) |  79.93($\pm$1.22) | 79.21($\pm$0.58) |  85.30($\pm$3.74) |  78.18($\pm$0.91) |  80.13($\pm$1.17)
WoLF+ | 81.09($\pm$0.30) | 81.60($\pm$1.56) |  79.20($\pm$0.55) | 86.22($\pm$3.62) | 78.84($\pm$1.89) | 81.39($\pm$1.65)

(a)
---

Comparison | Accuracy$\uparrow$ | Helpfulness$\uparrow$ | Relevance$\uparrow$ | Hallucination$\uparrow$ | Universality$\uparrow$
---|---|---|---|---|---
55pt. left model vs. right model | Gemini | PaLM-2 | Gemini | PaLM-2 | Gemini | PaLM-2 | Gemini | PaLM-2 | Gemini | PaLM-2
WoLF vs. LLaVA-Med [13] | 71.50% | 72.13% | 77.75% | 77.90% | 76.25% | 76.78% | 84.25% | 84.85% | 73.40% | 73.70%
WoLF vs. LLM-CXR [14] | 64.05% | 61.50% | 61.73% | 61.58% | 61.95% | 61.58% | 63.93% | 64.18% | 63.6% | 63.45%
WoLF vs. XrayGPT [24] | 75.80% | 75.83% | 83.68% | 83.15% | 81.05% | 82.05% | 84.23% | 84.68% | 84.93% | 84.63%
WoLF vs. WoLF- | 56.40% | 57.10% | 53.80% | 54.50% | 53.81% | 54.00% | 68.73% | 68.80% | 55.20% | 55.40%
WoLF vs.WoLF+ | 47.20% | 47.45% | 46.70% | 47.12% | 47.05% | 47.95% | 49.90% | 49.95% | 48.80% | 49.20%

(b)
---

Table 4: Accuracy of the CXR-VQA task by topic. We utilize the ELIXR [28]
framework for assessing this VQA performance in MIMIC-CXR [10].

Accuracy$\uparrow$ |  All$\uparrow$ |  Presence$\uparrow$ |  Location$\uparrow$ |  Size, severity, type$\uparrow$
---|---|---|---|---
ELIXR [28] |  54.8% |  64.5% |  41.0% |  25.0%
XrayGPT [24] |  25.2% |  27.4% |  21.9% |  20.3%
LLM-CXR [13] |  56.7% |  60.1% |  49.0% |  53.1%
MedXChat [29] |  61.2% |  61.5% |  56.3% |  68.8%
WoLF |  62.1% |  67.9% |  56.9% |  71.8%

Figure 6: Evaluation prompts for scoring (left) and prioritizing (right).
Preamble and evaluation criteria are common in both scoring and prioritizing
prompts. The two prompts, each for AI-evaluation (left) and for its ablation
on Win-rate (right), differ only in evaluation steps. Figure 7: Qualitative
results of visual question-answering scenarios. As shown in (a) and (b), the
model can be fed with patient histories and medication details from EHRs. WoLF
utilizes these external contexts to deliver more accurate responses. (c) shows
question-answering when there are no findings. The model answered correctly
that no disease could be found, without causing any hallucinations.