# Recent advancements of the experiment to search for $2K$-capture in 124Xe
using a Large Low-background Proportional Counter

O.D. Petrenko2 I.D. Fedorets2 A.M. Gangapshev1 Y.M. Gavrilyuk1 V.V. Kazalov1
V.V. Kuzminov1 S.I. Panasenko2 S.S. Ratkevich2 1 Baksan Neutrino Observatory
INR RAS, Russia  2 V.N.Karazin Kharkiv National University, Ukraine
<EMAIL_ADDRESS>
###### Abstract

A new method for correcting the charge signal from the large proportional
counter is proposed. The correction method makes it possible to take into
account the loss of primary ionization electrons in the presence of weakly
increasing microimpurities of electronegative gases during long-term
measurements. It allows increasing the efficiency of the selection of useful
events and thereby increasing the sensitivity to $2\nu 2K$-decay in 124Xe at
the level of $2\times 10^{22}$ from over six years of measurements using a
large proportional counter. Analysis of the calibration data, taking into
account the new method for correcting the energy release in the detector, made
it possible to register events from double $K$ photoionization by a single
photon.

##  1 Introduction

The experimental signature of the capture of two electrons from the inner
shell of 124Xe is the registration of two $K$-shell X-rays and an Auger
electron. Unambiguous selection of such a complex event is possible in
measurements using a large low background cylindrical proportional counter
(LPC). At the “$2K$-CAPTURE” facility of the DULB-4900 underground laboratory,
BNO INR RAS [1] has been carrying out a long experiment to search for double
beta decay processes in 124Xe for several years [2, 3]. A 10.37 L LPC made of
oxygen-free copper is used. The LPC is filled with a pure sample enriched to
21% in 124Xe at pressures up to 5.8 bar without any addition of quenching or
accelerating gas added.

The LPC signals were read out with a charge-sensitive amplifier (CSA) from one
end of the anode wire and recorded to the computer via a 50-MHz, 12-bit
waveform digitizer for later analysis. The pulse shape fully displayed the
information on the spatial distribution of the primary ionization charges
projected onto the counter radius. For more reliably extraction of useful
three-point events, in addition to good energy resolution, adequate measures
must be taken to determine the energy scale and resolution accurately and
monitor their stability over the full data acquisition period, and take into
consideration the relevant uncertainties entering the statistical analysis for
the $2K$-capture search.

##  2 Recovery of energy release for primary ionization in LPC

Figure 1:  Signal from the CSA: $(a)$ and $(c)$ \- the initial charge pulse
(dark curve) and the reconstructed signal in the offline mode (light curves);
$(b)$ and $(d)$ \- the differentiated charge pulse corresponding to the
reconstructed charge signal. $\tau_{p}$ is the time of electron drift from the
cathode to the anode. $(e)$ \- bell-shaped signal area (charge ) vs FWHM.
$(f)$ \- the rate of change degradation factor of the response amplitude from
$K$ of X-ray radiation vs the time in the run.

The shape of the electric current pulse is significantly affected by the
distribution of the density of primary ionization electrons crossing the
boundary of the gas amplification region. The distribution parameters depend
on the drift time of the electrons of the initial point charge to the anode.
The primary ionization cluster is blurred during drift due to the radial
diffusion of the electron cloud.

Figures 1$(a)$ and 1$(c)$ show pulses of three-point events, presumably caused
by a $2\nu 2K$-capture in 124Xe. Black curves represent recorded charge pulses
from CSA. The current pulses of the primary ionization electrons reconstructed
from the digitized charge signals at the boundary of the gas amplification
region are shown at Fig. 1 $(b)$ and $(d)$, the corresponding normalized
charge pulses obtained by integrating these current pulses are shown by
lighter color curves in Figs. 1$(a)$ and 1$(c)$. The elapsed time from the
onset of the primary pulse to the onset of the first afterpulse $(\tau_{p})$
equals the time of electron drift from the cathode to the anode. It presets
the duration of the time interval to allocate totally any single event
regardless of its primary ionization distribution over the LPC volume. In the
case of pure xenon the calculated drift time for the ionization electrons to
move from the cathode to the anode is 163 $\mu$s. The integral of the current
pulse in the case of a multipoint event for an interval $\tau_{p}$, starting
from the beginning of the pulse, gives the total number of the primary
ionization electrons. A detailed description of the separation method of the
electronic and ionic component of the primary ionization cloud from the
digitized charge pulse and further extraction of point ionization in a
multipoint event is given in Ref. [4].

A set of bell-shaped curves can describe the resulting current waveform [Fig.
1$(b)$ and 1$(d)$], which allows to determine the charge (energy) released in
every point ionization cluster of multipoint event. The diffusion of primary
electrons in the drift region directly affects the distribution of their
arrival times in the anode region of high fields. Thus, one can distinguish
between events near the cathode surface associated with a long rise time of
the charge pulse (wide bell-shaped distributions of the current signal) and
events in the internal volume with a shorter rise time (small widths of the
current signal).

As an internal detector calibration, we used the photoionization effect by
background photons to the $K$-shell of the working gas atoms in the case of
$K$ fluorescence. The two-dimensional distribution of individual amplitudes
selected from two-point events with a total energy of 1-200 keV as a function
of the pulse FWHM (rise time) is shown in Fig. 1$(e)$. The locus of
photoionization to the Xe-$K$-shell is clearly visible. This locus makes it
possible to determine the degree of degradation of the primary ionization
charge as a function of the pulse rise time. A charge loss correction factor
can be obtained by analyzing this relationship for a set of two-week
background data collected between two cleanings of gas in the Ti reactor. In
this way, the amplitude of each detector response can is corrected. The rate
of change in the degradation coefficient of the detector response to the
$K$-series X-rays is shown in Fig. 1$(f)$.

##  3 Precision analysis of the LPC response to signals from multipoint
ionization

It is known that, besides the $2K$ capture, a double-$K$-shell photoionization
of the atom can create the “hollow atom” by absorbing a single photon with the
release of two correlated $K$ electrons. This background events can be used as
reference in $2K$-capture experiments, because background $\gamma$ rays are
absorbed there within the fiducial volume of the counter. In order to select
such events, the entire set of calibration measurements from an external
$\gamma$-source (see Ref. [5]) accumulated over six years was reanalyzed to
test the procedures for processing background signals in the experiment.
Additionally, the calibration data were analyzed when filling the LPC with
krypton to a pressure of 5.5 bar. During calibration, the counter was
irradiated through the casing wall by photons from the 109Cd source with
energy 88 keV. In addition to regular calibration measurements, long-term
measurements were performed over several months.

Figure 2:  Two-dimensional distributions of the reconstructed amplitudes of
energy release in individual point ionization clusters from an external source
of 88-keV photons (109Cd): top panels - two-point events and bottom panels -
three-point events for krypton and xenon samples, respectively.

To confirm our assumptions the characteristics of two-point events with a
total amplitude of 60-95 keV, Fig. 2 (top panels) shows a two-dimensional
distribution of these events depending on the amplitude of the first (A1) and
second (A2) subpulses for krypton (left) and xenon (right). It can be seen
from this distribution that intensity maxima are observed for a combination of
subpulse amplitudes [$K$-X-ray and photoelectron + Auger electron]. The
vertical and horizontal loci reflect the features of events from incomplete
absorption of 88 keV energy and the shape of the input radiation spectrum. The
oblique diagonal band includes events with total absorption of 88 keV by the
photoelectric effect with the production of bremsstrahlung photons and via
Compton scattering. Two oblique diagonal stripes are visible for the xenon
target. One band includes events with total absorption of an 88-keV photon by
the photoelectric effect with emission $e_{A}(K)$ and with the formation of
bremsstrahlung photons and through Compton scattering. The second band
corresponds to an energy of the 58.2 keV and includes events from the peak
emission of 29.8 keV from Compton scattering of 88-keV photon by external
electrons. The procedure for fitting a flat substrate into two-dimensional
distributions avoids the problem of the Compton background and has the
advantage that errors are determined fitting procedure.

The bottom panels of Fig. 2 show two-dimensional distribution of three-point
events when convolving in terms of the amplitude of the first subpulse. For a
krypton target, pronounced bunches of points are observed at combinations of
amplitudes [A2,A3] corresponding to [$K_{\alpha}^{h}\otimes K_{\alpha}^{s}$]
and [$K_{\alpha}$ $\otimes$ $(\gamma 88-2\times K_{\alpha}$)]. For correct
explanation of the observed pattern, it should be borne in mind that all three
subpulses have identical amplitude spectra. The first region includes events
in which the amplitudes of the second and third subpulses correspond to the
energy of the $K$-series of a krypton atom. The invisible first subpulse A1
has an amplitude corresponding to energy 62.8 keV and complements the total
energy of the event 88 keV. The corresponding event selection for the xenon
target is shown in the bottom right panel of the same figure.

Summing the corrected output for each run gives the $I_{1K}(\alpha,\beta)$
value for the normal lines of the $K$-series diagram to obtain the integrated
intensity of the dominant single $K$-shell photoionization. Similarly, sums
were carried out on the spectra for three-point events to yield the integrated
intensity $I_{2K}(\alpha,\beta)$ for double $K$-shell photoionization. The
total number of registered events was $2.5\times 10{{}^{7}}$ and $4.2\times
10{{}^{7}}$ for krypton and xenon, respectively. The estimate of the total
number of double $K$-shell photoionizations taking

Figure 3:  The ratio of double to single $K$-shell photoionization as function
of atomic number. Solid symbols are used for experiments carried out in the
asymptotic region for the cross section of double photoionization [6]. Solid
line is the power-law $1/Z^{1.61}$ fit.

into account the fluorescence yield was 550 and 320 events for krypton and
xenon, respectively. The lower yield for the xenon target can be explained by
the fact that the energy of the incident photons was lower than the peak
value, in contrast to the krypton target for which $\Delta
E=E_{\gamma}-Q_{2K}$ located in the asymptotic region.

Thus, due to the data obtained, we can estimate the ratio of double and single
ionization of $K$-shells by a single photon with an energy of 88 keV for
shake-off processes, which was $5.2(3)\times 10^{-4}$ and $1.2(7)\times
10^{-4}$ for krypton and xenon, respectively.

Figure 3 shows the ratio of the probability of double- to single-
photoionization of the $K$-shell as a function of the atomic number. Black
symbols are used for data of experiments carried out in the asymptotic region
for the double-$K$-photoionization cross-section [6]. Red dots show our data
obtained.

##  4 Conclusions

This work demonstrates the testing of a new method for correcting the energy
release in multi-cluster ionization in the LPC. For this, the detector was
continuously calibrated for several thousand hours from an external collimated
109Cd source filled with both krypton and xenon. Using this technique, we
predict an increase in the sensitivity to $2\nu 2K$-decay in 124Xe of $2\times
10^{22}$ yr after six years of operation in the current experiment.

## References

## References

  * [1] Gavriljuk Ju M _et al._ 2013 _Nucl. Instr. Meth. A._ 729 576
  * [2] Gavriljuk Yu M _et al._ 2015 _Physics of Atomic Nuclei_ 78 1563
  * [3] Gavriljuk Y M _et al._ 2018 _Physics of Particles and Nuclei_ 49 4 563
  * [4] Gavrilyuk Y M _et al._ 2010 _Instrum. Exp. Tech._ 53 57; 2013 _Phys. Rev. C_ 87 035501
  * [5] Petrenko O D _et al._ 2020 J. Phys.: Conf. Series 1690 012037
  * [6] Kanter E _et al._ 2006 _Phys. Rev. A_ 73 022708


# CoF-CoT: Enhancing Large Language Models with Coarse-to-Fine Chain-of-
Thought Prompting for Multi-domain NLU Tasks

Hoang H. Nguyen1, Ye Liu2, Chenwei Zhang3, Tao Zhang1, Philip S. Yu1
1 Department of Computer Science, University of Illinois at Chicago, Chicago,
IL, USA
2 Salesforce Research, Palo Alto, CA, USA
3 Amazon, Seattle, WA, USA
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

While Chain-of-Thought prompting is popular in reasoning tasks, its
application to Large Language Models (LLMs) in Natural Language Understanding
(NLU) is under-explored. Motivated by multi-step reasoning of LLMs, we propose
Coarse-to-Fine Chain-of-Thought (CoF-CoT) approach that breaks down NLU tasks
into multiple reasoning steps where LLMs can learn to acquire and leverage
essential concepts to solve tasks from different granularities. Moreover, we
propose leveraging semantic-based Abstract Meaning Representation (AMR)
structured knowledge as an intermediate step to capture the nuances and
diverse structures of utterances, and to understand connections between their
varying levels of granularity. Our proposed approach is demonstrated effective
in assisting the LLMs adapt to the multi-grained NLU tasks under both zero-
shot and few-shot multi-domain settings 111https://github.com/nhhoang96/CoF-
CoT.

##  1 Introduction

Natural Language Understanding (NLU) of Dialogue systems encompasses tasks
from different granularities. Specifically, while intent detection requires
understanding of coarse-grained sentence-level semantics, slot filling
requires fine-grained token-level understanding. Moreover, Semantic Parsing
entails the comprehension of connections between both token-level and
sentence-level tasks.

Large Language Models (LLMs) possess logical reasoning capability and have
yielded exceptional performance Zoph et al. (2022); Zhao et al. (2023b).
However, they remain mostly restricted to reasoning tasks. On the other hand,
mutli-step reasoning can take place when solving multiple interconnected tasks
in a sequential order. In practical NLU systems, as coarse-grained tasks are
less challenging, they can be solved first before proceeding to fine-grained
tasks. Therefore, the coarse-grained tasks’ outcomes can provide valuable
guidance towards subsequent fine-grained tasks, allowing for deeper semantic
understanding of diverse utterances across different domains within NLU
systems Firdaus et al. (2019); Weld et al. (2022); Nguyen et al. (2023a). For
instance, consider the utterance “Remind John the meeting time at 8am” under
reminder domain, recognizing GET_REMINDER_DATE_TIME intent is crucial for
correctly understanding the existence of PERSON_REMINDED slot type rather than
CONTACT or ATTENDEE slot type.

Figure 1: Illustration of Abstract Meaning Representation (AMR) of two
structurally different but semantically similar utterances with the same fine-
grained and coarse-grained labels. Each colored node represents an AMR concept
matching the colored word or phrase existent in the corresponding utterances.
Figure 2: Illustration of CoF-CoT and its counterpart Direct Prompt approach.
The left side illustrates the proposed CoF-CoT. The right side illustrates the
naive Direct Prompt approach. Red and Green represent sentence-level and
token-level annotations captured in the Logic Form respectively. For CoF-CoT,
the prompt at each step starting from Step 2 is conditioned on the relevant
output predicted from the previous step(s).

Chain-of-Thought (CoT) Wei et al. (2022) provides an intuitive approach to
elicit multi-step reasoning from LLMs automatically. However, there remain two
major challenges with the current CoT approach: (1) LLMs entirely rely on
their uncontrollable pre-trained knowledge to generate step-by-step reasoning
and could result in unexpected hallucinations Yao et al. (2022); Zhao et al.
(2023a), (2) Additional beneficial structured knowledge cannot be injected
into LLMs via the current CoT.

On the other hand, structured representation demonstrates the effectiveness in
enhancing the capability of Pre-trained Language Models (PLMs) Xu et al.
(2021); Bai et al. (2021); Shou et al. (2022). In Dialogue systems, the
dependencies among different dialogue elements together with the existent
diversely structured utterances necessitate the integration of additional
structured representation. For instance, as observed in Figure 1, by
leveraging Abstract Meaning Representation (AMR) Banarescu et al. (2013), it
is possible to map multiple semantically similar but structurally different
utterances with similar coarse-grained and fine-grained labels into the same
structured representation, allowing for effective extraction of intents,
slots, and their interconnections within the Dialogue systems.

In our work, we explore the capability of LLMs in NLU tasks from various
granularities, namely multi-grained NLU tasks. Motivated by CoT, we propose an
adaptation of CoT in solving multi-grained NLU tasks with an integration of
structured knowledge from AMR Graph. Our contribution can be summarized as
follows:

$\bullet$ To the best of our knowledge, we conduct the first preliminary study
of LLMs’ capability in multi-grained NLU tasks of the Dialogue systems.
$\bullet$ We propose leveraging a CoT-based approach to solve multi-grained
NLU tasks in a coarse-to-fine-grained sequential reasoning order.
$\bullet$ We propose integrating structured knowledge represented via AMR
Graph in the multi-step reasoning to capture the shared semantics across
diverse utterances within the Dialogue systems.

##  2 Related Work

#### Chain-of-Thought (CoT)

CoT Wei et al. (2022) proposes leveraging intermediate steps to extracts
logical reasoning of LLMs and succeeds in various reasoning tasks. Wang et al.
(2022) enhances CoT by selecting the most consistent output answers via
majority voting. Additionally, Fu et al. (2022) argues majority consistency
voting works best among the most complex outputs. They propose complexity
metrics and leverage them to select demonstration samples and decoding
outputs. Unlike previous CoT approaches, we leverage CoT to solve multi-
grained NLU tasks.

#### Structured Representation

Structured Representation has been widely incorporated in language models to
further enhance the capability across various NLP tasks Bugliarello and
Okazaki (2020); Zhang et al. (2020). Structured representation can be either
in the syntax-based structure Bai et al. (2021); Xu et al. (2021) such as
Dependency Parsing (DP) Graph, Constituency Parsing (CP) Graph or semantic-
based structure Shou et al. (2022) such as Abstract Meaning Representation
(AMR) Graph Banarescu et al. (2013). Unlike previous works, we aim at
leveraging structured representation as an intermediate step in the multi-step
reasoning approach to extract essential concepts from diverse utterances in
the multi-domain Dialogue systems.

Table 1: Experimental results on MTOP and MASSIVE under zero-shot and few-shot
multi-domain settings.

MTOP
---
Model | Zero-shot | Few-shot
| NLU | Semantic Parsing | NLU | Semantic Parsing
| Intent Acc | Slot F1 | Frame Acc | Exact Match | Intent Acc | Slot F1 | Frame Acc | Exact Match
Direct Prompt | 31.50 $\pm$ 1.80 | 21.84 $\pm$ 2.83 | 8.33 $\pm$ 1.44 | 6.00 $\pm$ 1.32 | 51.33 $\pm$ 3.40 | 28.35 $\pm$ 3.24 | 11.00 $\pm$ 1.80 | 8.33 $\pm$ 1.00
CoT | 31.83 $\pm$ 2.02 | 22.40 $\pm$ 1.61 | 8.67 $\pm$ 0.35 | 6.33 $\pm$ 1.04 | 47.67 $\pm$ 5.20 | 28.46 $\pm$ 3.10 | 11.83 $\pm$ 1.53 | 8.50 $\pm$ 1.04
SC-CoT | 32.50 $\pm$ 1.89 | 22.71 $\pm$ 2.44 | 10.05 $\pm$ 0.87 | 6.83 $\pm$ 0.76 | 53.50 $\pm$ 3.04 | 29.53 $\pm$ 1.99 | 12.50 $\pm$ 1.80 | 9.00 $\pm$ 0.87
ComplexCoT | 32.67 $\pm$ 2.00 | 22.86 $\pm$ 3.17 | 10.83 $\pm$ 0.29 | 7.16 $\pm$ 0.58 | 48.83 $\pm$ 2.47 | 29.21 $\pm$ 2.65 | 13.17 $\pm$ 0.58 | 8.83 $\pm$ 2.89
Least-to-Most | 45.67 $\pm$ 0.58 | 21.84 $\pm$ 1.91 | 14.50 $\pm$ 0.50 | 8.00 $\pm$ 0.50 | 49.83 $\pm$ 4.54 | 27.28 $\pm$ 2.41 | 16.00 $\pm$ 0.50 | 8.83 $\pm$ 0.76
Plan-and-Solve | 45.00 $\pm$ 4.00 | 22.45 $\pm$ 2.28 | 9.50 $\pm$ 1.61 | 8.25 $\pm$ 2.25 | – | – | – | –
CoF-CoT | 57.67 $\pm$ 2.75 | 23.47 $\pm$ 4.09 | 14.33 $\pm$ 1.52 | 9.00 $\pm$ 1.00 | 61.50 $\pm$ 4.93 | 30.12 $\pm$ 3.93 | 15.00 $\pm$ 1.32 | 11.00 $\pm$ 1.61

MASSIVE
---
Direct Prompt | 72.50 $\pm$ 4.58 | 33.24 $\pm$ 3.34 | 24.17 $\pm$ 3.79 | 20.67 $\pm$ 3.28 | 75.17 $\pm$ 0.58 | 42.36 $\pm$ 2.98 | 29.00 $\pm$5.39 | 24.50 $\pm$ 4.07
CoT | 71.83 $\pm$ 2.57 | 36.32 $\pm$ 1.94 | 24.50 $\pm$ 2.29 | 21.66 $\pm$ 3.40 | 76.83 $\pm$ 3.82 | 44.89 $\pm$ 2.50 | 31.33 $\pm$ 0.87 | 25.83 $\pm$ 2.25
SC-CoT | 73.05 $\pm$ 1.27 | 37.06 $\pm$ 2.54 | 27.16 $\pm$3.21 | 22.50 $\pm$ 2.65 | 77.33 $\pm$ 2.89 | 47.02 $\pm$ 4.60 | 34.00 $\pm$ 3.21 | 27.16 $\pm$ 3.50
ComplexCoT | 73.66 $\pm$ 3.65 | 37.64 $\pm$ 3.51 | 25.83 $\pm$ 2.25 | 22.16 $\pm$ 2.51 | 77.83 $\pm$ 1.83 | 46.59 $\pm$ 2.43 | 36.50 $\pm$ 2.89 | 28.00 $\pm$ 3.69
Least-to-Most | 72..83 $\pm$ 4.65 | 37.62 $\pm$ 1.69 | 31.50 $\pm$ 1.53 | 26.50 $\pm$ 1.26 | 77..00 $\pm$ 3.28 | 45.93 $\pm$ 3.99 | 32.50 $\pm$ 4.09 | 29.00 $\pm$ 5.11
Plan-and-Solve | 69.33 $\pm$ 2.47 | 38.07 $\pm$ 2.07 | 32.00 $\pm$ 1.26 | 29.00 $\pm$ 1.26 | – | – | – | –
CoF-CoT | 89.00 $\pm$ 2.29 | 38.66 $\pm$ 3.25 | 33.17 $\pm$ 4.04 | 25.50 $\pm$ 2.64 | 92.00 $\pm$2.29 | 47.06 $\pm$ 4.63 | 37.50 $\pm$ 1.89 | 29.50 $\pm$ 3.12

##  3 Proposed Framework

In this section, we introduce our proposed Coarse-to-Fine Chain-of-thought
(CoF-CoT) approach for NLU tasks as depicted in Figure 2. Specifically, we
propose a breakdown of multi-grained NLU tasks into 5 sequential steps from
coarse-grained to fine-grained tasks. At each step, LLMs leverage the
information from the previous steps as a guidance towards the current
predictions. As domain name could provide guidance to NLU tasks Xie et al.
(2022); Zhou et al. (2023), at each step, we condition the domain name of the
given utterance in the input prompt. The model’s output is in the format of
Logic Form Kamath and Das  which encapsulates coarse-grained intent label,
fine-grained slot labels and slot values. Further details of Logic Form’s
structure and its connections with multi-grained NLU tasks are provided in the
Appendix A.

Our multi-step reasoning is designed in the following sequential order:

1\. Generate AMR: Given the input utterance, LLMs generate the AMR structured
representation Banarescu et al. (2013). The representation is preserved in the
Neo-Davidsonian format as demonstrated in Figure 1,2. Each node in AMR graph
refers to a concept, including entity, noun phrase, pre-defined frameset or
special keyword. Edges connecting two nodes represent the relation types.

2\. Generate Intent: In this step, LLMs generate coarse-grained intent label
prediction when conditioned on the given input and its corresponding AMR
Graph. AMR concepts could provide additional contexts to ambiguous utterances,
leading to improved ability to recognize the correct intents.

3\. Generate Slot Values: In this stage, to generate the fine-grained slot
values existent in the input utterance, besides the utterance itself, prompts
for LLMs are conditioned on the generated AMR structure and predicted intent
label. As AMR graph captures the essential concepts existent in the utterance
while abstracting away syntactic idiosyncrasies of the utterance, it can help
extract the important concepts mentioned in the utterances. In order to
further couple the connections between slots and intents Zhang et al. (2019);
Wu et al. (2020), predicted intents from the Step 2 are also concatenated to
construct input prompts for Step 3.

4\. Generate Slot Value, Slot Type pairs: After obtaining slot values, LLMs
label each identified slot value when given the slot vocabulary. Similar to
Step 3, we condition the generated output with the predictions from previous
steps, including AMR and intent. Both AMR and intent provide additional
contexts for slot type predictions of the given slot values besides the input
utterance.

5\. Generate Logic Form: The last step involves aggregating the predicted
intents together with sequences of slot type and slot value pairs to construct
the final Logic Form predictions.

##  4 Experiments

###  4.1 Datasets & Preprocessing

We evaluate our proposed framework on two multi-domain NLU datasets, namely
MTOP Li et al. (2021) and MASSIVE Bastianelli et al. (2020); FitzGerald et al.
(2022). As the innate capability of language understanding is best represented
via the robustness across different domains, we evaluate the frameworks under
low-resource multi-domain settings, including zero-shot and few-shot. Details
of both datasets are provided in Appendix B.

To provide a comprehensive evaluation for coarse-grained, fine-grained NLU
tasks, as well as the interactions between the two, we conduct an extensive
study on both NLU and Semantic Parsing metrics, including: Slot F1-score,
Intent Accuracy, Frame Accuracy, Exact Match. Intent Accuracy assesses the
performance on coarse-grained sentence-level tasks, while Slot F1 metric
evaluates the performance on more fine-grained token-level tasks. The
computation of Frame Accuracy and Exact Match captures the ability to
establish the accurate connections between sentence-level and token-level
elements. For more details of individual metric computation from the Logic
Form, we refer readers to Li et al. (2021).

To conduct the evaluation with efficient API calls, following Khattab et al.
(2022), we construct test sets by randomly sampling 200 examples covering a
set of selected domains, namely test domains. We repeat the process with 3
different seeds to generate 3 corresponding test sets. Reported performance is
the average across 3 different seed test sets with standard deviations. For
few-shot settings, we randomly select a fixed k samples from a disjoint set of
domains, namely train domains. These samples are manually annotated with
individual step labels as commonly conducted by other in-context learning CoT
approaches Wei et al. (2022); Wang et al. (2022). Additional implementation
details as well as the prompt design and sample outputs are provided in
Appendix C and D respectively.

###  4.2 Baseline

#### Chain-of-Thought (CoT) Approach Comparison

We compare our proposed method with the current relevant state-of-the-art CoT
approaches:

$\bullet$ Direct Prompt: Naive prompting to generate the Logic Form given the
intent and slot vocabulary.
$\bullet$ CoT Wei et al. (2022): Automatic generation of series of
intermediate reasoning steps from LLMs
$\bullet$ SC-CoT Wang et al. (2022): Enhanced CoT via majority voting among
multiple reasoning paths.
$\bullet$ Complex-CoT Fu et al. (2022) : Enhanced CoT by selecting and
measuring the consistency of the most complex samples. In our case, we
leverage the longest output as the complexity measure.
$\bullet$ Least-to-Most Zhou et al. (2022) : Enhanced CoT by first
automatically decomposing the in-hand problems into series of simpler sub-
problems, and then solving each sub-problem sequentially.

$\bullet$ Plan-and-Solve Wang et al. (2023) : Enhanced CoT by guiding LLMs to
devise the plan before solving the problems by prompting “Let’s first
understand the problem and devise a plan to solve the problem. Then, let’s
carry out the plan and solve the problem step by step.”

#### Fine-tuning (FT) Approach Comparison

As one of the early studies in leveraging LLM for NLU tasks, we also conduct
additional comparisons with traditional FT approaches. Specifically, we
leverage RoBERTa PLM Zhuang et al. (2021) with joint Slot Filling and Intent
Detection objectives Li et al. (2021) as the FT model. Unlike LLM, traditional
FT operates under closed-world assumption which requires sufficient data to
learn domain-specific and domain-agnostic feature extraction in multi-domain
settings. For a fair comparison with LLM, we impose an essential constraint
that there exist no overlapping domains between train and test domains under
ZSL and FSL setting for both FT and CoT approaches. This leads to 3 different
scenarios for FT approaches, including:

$\bullet$ Fully Supervised: Samples sharing similar domains with test sets are
used for training.
$\bullet$ ZSL: We utilize samples from domains different from test domains for
training.
$\bullet$ FSL: We leverage samples from domains different from test domains in
conjunction with a fixed number of k-shot test domain samples.

Table 2: Comparison between FT and LLM approaches on MTOP dataset.

Method | Assumption | Intent Acc | Slot F1 | Frame Acc | Exact Match
---|---|---|---|---|---
RoBERTa FT | Supervised | 67.19 $\pm$ 2.90 | 75.17 $\pm$ 1.08 | 43.57$\pm$ 4.18 | 36.10 $\pm$ 1.08
RoBERTa FT | ZSL | 0 | 12.68 $\pm$ 1.25 | 0 | 0
RoBERTa FT | FSL | 0 | 13.75 $\pm$ 1.22 | 0 | 0
CoF-CoT | ZSL | 57.67 $\pm$ 2.75 | 23.47 $\pm$ 4.09 | 14.33 $\pm$ 1.52 | 9.00 $\pm$ 1.00
CoF-CoT | FSL | 61.50 $\pm$ 4.93 | 30.12 $\pm$ 3.93 | 15.00 $\pm$ 1.32 | 11.00 $\pm$ 1.61

Table 3: Ablation study on the effectiveness of different structured
representations on MTOP dataset under zero-shot settings. CP, DP, AMR denote
Constituency Parsing, Dependency Parsing and Abstract Meaning Representation
respectively.

| Intent Acc | Slot F1 | Frame Acc | Exact Match
---|---|---|---|---
CoT (w/o structure) | 57.16 $\pm$ 3.69 | 17.50 $\pm$ 2.92 | 12.16 $\pm$ 1.61 | 4.67 $\pm$ 3.33
CP-CoT | 57.33 $\pm$ 3.25 | 19.34 $\pm$ 3.34 | 13.16 $\pm$ 1.04 | 5.50 $\pm$ 1.32
DP-CoT | 57.50 $\pm$ 3.01 | 17.83 $\pm$ 2.53 | 12.67 $\pm$ 1.04 | 5.83 $\pm$ 2.08
AMR-CoT | 57.67 $\pm$ 2.75 | 23.47 $\pm$ 4.09 | 14.33 $\pm$ 1.52 | 9.00 $\pm$ 1.00

Table 4: Ablation study of step ordering on MASSIVE dataset. CoF and FoC
denote Coarse-to-Fine-grained and Fine-to-Coarse-grained order respectively.

Method | Assumption | Intent Acc | Slot F1 | Frame Acc | Exact Match
---|---|---|---|---|---
Random-CoT | Random Order | 80.67 $\pm$3.60 | 27.14 $\pm$ 2.47 | 26.50 $\pm$1.80 | 16.50 $\pm$ 1.04
FoC-CoT | FoC order | 83.00 $\pm$ 2.88 | 32.11 $\pm$ 2.50 | 28.50 $\pm$ 3.21 | 18.00 $\pm$ 3.50
CoF-CoT (w/o step 1) | No AMR | 81.50 $\pm$ 4.36 | 33.68 $\pm$ 2.40 | 27.50 $\pm$ 2.65 | 18.00 $\pm$ 0.76
CoF-CoT (w/o step 2) | No intent | 78.17 $\pm$ 4.80 | 27.66 $\pm$ 1.93 | 23.50 $\pm$ 2.78 | 14.50 $\pm$ 2.25
CoF-CoT (w/o step 3) | No separate KP | 82.33 $\pm$ 1.04 | 34.63 $\pm$ 3.10 | 32.83 $\pm$ 2.47 | 23.00 $\pm$ 1.80
CoF-CoT (w/o step 4) | No separate slot prediction for KP | 79.17 $\pm$ 4.01 | 32.92 $\pm$ 5.02 | 31.50 $\pm$ 3.50 | 21.83 $\pm$ 3.17
CoF-CoT (w/o step 3+4) | No separate slot prediction | 81.33 $\pm$ 4.19 | 31.31 $\pm$3.77 | 27.67 $\pm$ 5.34 | 21.00 $\pm$ 4.92
CoF-CoT | CoF order (Full) | 89.00 $\pm$ 2.29 | 38.66 $\pm$ 3.25 | 33.17 $\pm$ 4.04 | 25.50 $\pm$ 2.64
\- Conditioning | No domain | 84.50 $\pm$ 2.75 | 36.80 $\pm$ 2.08 | 32.50 $\pm$ 1.73 | 24.83 $\pm$ 0.58

##  5 Result & Discussion

As observed in Table 1, our proposed CoF-CoT achieves state-of-the-art
performance across different evaluation metrics on MASSIVE and MTOP datasets
under both zero-shot and few-shot settings. The performance gain over the most
competitive baseline is more significant in terms of Intent Accuracy (25% and
15.34% improvements on MTOP and MASSIVE respectively in zero-shot settings).
Additional case studies presented in Appendix E further demonstrate the
effectiveness of CoF-CoT.

In addition, we observe consistent improvements of different CoT variants over
the Direct Prompt. It implies that CoT prompting allows the model to reason
over multiple steps and learn the connections between different NLU tasks more
effectively.

In comparison with MASSIVE, performance of all methods is significantly lower
on MTOP. It is mainly due to the more complex Logic Form structures existent
in MTOP. It is noticeable that MASSIVE datasets contain samples of fewer
average number of slots, leading to significantly better performance on
Semantic Parsing tasks (i.e. Frame Accuracy and Exact Match).

Our CoF-CoT shares certain degrees of similarities with Least-to-Most Zhou et
al. (2022), Plan-and-Solve prompting Wang et al. (2023). However, unlike the
two aforementioned baselines that rely heavily on the existent pre-trained
knowledge of LLMs, CoF-CoT provides a controllable number of sequential steps
and conditioning inputs for each step, allowing for flexible adaptations and
customizations to future downstream tasks that LLMs might not be familiar
with.

#### Comparison with FT

Under ZSL and FSL settings, the FT model suffers from the aforementioned
domain gap issues. Specifically, as observed in Table 2, since there exist
minimal overlapping intent labels between train and test domains, without
sufficient data in ZSL and FSL settings, the FT approaches are unable to learn
transferable multi-domain features, leading to 0 performance in Intent
Accuracy. This behavior also results in 0 performance for both Frame Acc and
Exact Match as the correct intents are the prerequisites for correct semantic
frame and exact match metrics. On the other hand, Fully supervised FT approach
acquires domain-specific knowledge of target domains from training data and
performs the best across different evaluation metrics. However, this
assumption does not directly match ZSL/FSL settings in which LLMs are
currently evaluated.

#### Impact of Structured Representation

Besides AMR Graph, there exist other structured representations that directly
link to semantic and syntactic understanding of utterances, including DP,CP.
Our empirical study presented in Table 3 reveals that AMR-CoT unanimously
achieves the best performance, demonstrating its effectiveness in capturing
the diversity of input utterances when compared with other structured
representations.

#### Impact of Step Order

To further understand the importance of the designed CoF step order, we
conduct additional ablation studies on 3 different scenarios: (1) random
ordering (step 3$\rightarrow$1$\rightarrow$2$\rightarrow$4$\rightarrow$5), (2)
Fine-to-Coarse (FoC) ordering (step
1$\rightarrow$3$\rightarrow$4$\rightarrow$2$\rightarrow$5), and (3) CoF
ordering with hypothetical individual step removal. Table 4 demonstrates CoF
logical ordering yields the best performance with significant improvements on
the challenging Exact Match metrics (9.00 and 7.50 points of improvement over
random and FoC respectively). Random, FoC ordering together with CoF ordering
with missing individual steps neglect the natural connections of problem-
solving from high-level (coarse-grained) to low-level (fine-grained) tasks,
leading to worse performance across different metrics. For CoF-CoT, when step
1 or step 2 is removed (no AMR or intent information), we observe the most
significant performance decrease, implying the essence of coarse-grained
knowledge for LLMs to solve the later sequential steps.

#### Impact of Conditioning

The major advantage of our multi-step reasoning is the ability to explicitly
condition the prior predictions in later steps. As observed in Table 4,
conditioning prior knowledge in multi-step reasoning improves the overall
performance of CoF-CoT across different metrics with the most significant gain
in Intent Accuracy (+4.50%). This observation implies the importance of
conditioning the appropriate information on CoT for an improved performance of
LLMs under challenging zero-shot multi-domain settings.

##  6 Conclusion

In this work, we conduct a preliminary study of LLMs’ capability in multi-
grained NLU tasks of Dialogue systems. Moreover, motivated by CoT, we propose
a novel CoF-CoT approach aiming to break down NLU tasks into multiple
reasoning steps where (1) LLMs can learn to acquire and leverage concepts from
different granularities of NLU tasks, (2) additional AMR structured
representation can be integrated and leveraged throughout the multi-step
reasoning. We empirically demonstrate the effectiveness of CoF-CoT in
improving LLMs capability in multi-grained NLU tasks under both zero-shot and
few-shot multi-domain settings.

## Limitations

Our empirical study is restricted to English NLU data. It is partially due to
the existent English-bias of Abstract Meaning Representation (AMR) structure
Banarescu et al. (2013). We leave the adaptation of the CoF-CoT to
multilingual settings Nguyen and Rohrbaugh (2019); Qin et al. (2022); Nguyen
et al. (2023b) as future directions for our work.

Our work is empirically studied on the Flat Logic Form representation. In
other words, Logic Form only includes one intent followed by a set of slot
sequences. There are two major rationales for our empirical scope. Firstly, as
the early preliminary study on multi-grained NLU tasks which unify both
Semantic Parsing and NLU perspectives, we design a small and controllable
scope for the experiments. Secondly, as most NLU datasets including MASSIVE
FitzGerald et al. (2022) are restricted to single-intent utterances, Flat
Logic Form is a viable candidate reconciliating between traditional NLU and
Semantic Parsing evaluations. We leave explorations on the more challenging
Nested Logic Form where utterances might contain multiple intents for future
work.

##  7 Acknowledgement

We thank the anonymous reviewers for their constructive feedback which we
incorporated in the final version of this manuscript.

This work is supported in part by NSF under grant III-2106758.

## References

  * Bai et al. (2021) Jiangang Bai, Yujing Wang, Yiren Chen, Yaming Yang, Jing Bai, Jing Yu, and Yunhai Tong. 2021.  Syntax-BERT: Improving pre-trained transformers with syntax trees.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 3011–3020, Online. Association for Computational Linguistics.
  * Banarescu et al. (2013) Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013.  Abstract meaning representation for sembanking.  In _Proceedings of the 7th linguistic annotation workshop and interoperability with discourse_ , pages 178–186.
  * Bastianelli et al. (2020) Emanuele Bastianelli, Andrea Vanzo, Pawel Swietojanski, and Verena Rieser. 2020.  SLURP: A spoken language understanding resource package.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7252–7262, Online. Association for Computational Linguistics.
  * Bugliarello and Okazaki (2020) Emanuele Bugliarello and Naoaki Okazaki. 2020.  Enhancing machine translation with dependency-aware self-attention.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1618–1627, Online. Association for Computational Linguistics.
  * Casanueva et al. (2022) Iñigo Casanueva, Ivan Vulić, Georgios Spithourakis, and Paweł Budzianowski. 2022.  Nlu++: A multi-label, slot-rich, generalisable dataset for natural language understanding in task-oriented dialogue.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 1998–2013.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_.
  * Firdaus et al. (2019) Mauajama Firdaus, Ankit Kumar, Asif Ekbal, and Pushpak Bhattacharyya. 2019.  A multi-task hierarchical approach for intent detection and slot filling.  _Knowledge-Based Systems_ , 183:104846.
  * FitzGerald et al. (2022) Jack FitzGerald, Christopher Hench, Charith Peris, Scott Mackie, Kay Rottmann, Ana Sanchez, Aaron Nash, Liam Urbach, Vishesh Kakarala, Richa Singh, Swetha Ranganath, Laurie Crist, Misha Britan, Wouter Leeuwis, Gokhan Tur, and Prem Natarajan. 2022.  Massive: A 1m-example multilingual natural language understanding dataset with 51 typologically-diverse languages.
  * Fu et al. (2022) Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. 2022.  Complexity-based prompting for multi-step reasoning.  _arXiv preprint arXiv:2210.00720_.
  * (10) Aishwarya Kamath and Rajarshi Das.  A survey on semantic parsing.  In _Automated Knowledge Base Construction (AKBC)_.
  * Khattab et al. (2022) Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. 2022.  Demonstrate-search-predict: Composing retrieval and language models for knowledge-intensive nlp.  _arXiv preprint arXiv:2212.14024_.
  * Li et al. (2021) Haoran Li, Abhinav Arora, Shuohui Chen, Anchit Gupta, Sonal Gupta, and Yashar Mehdad. 2021.  MTOP: A comprehensive multilingual task-oriented semantic parsing benchmark.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 2950–2962, Online. Association for Computational Linguistics.
  * Nguyen and Rohrbaugh (2019) Hoang Nguyen and Gene Rohrbaugh. 2019.  Cross-lingual genre classification using linguistic groupings.  _Journal of Computing Sciences in Colleges_ , 34(3):91–96.
  * Nguyen et al. (2023a) Hoang Nguyen, Chenwei Zhang, Ye Liu, and Philip Yu. 2023a.  Slot induction via pre-trained language model probing and multi-level contrastive learning.  In _Proceedings of the 24th Meeting of the Special Interest Group on Discourse and Dialogue_ , pages 470–481, Prague, Czechia. Association for Computational Linguistics.
  * Nguyen et al. (2020) Hoang Nguyen, Chenwei Zhang, Congying Xia, and S Yu Philip. 2020.  Dynamic semantic matching and aggregation network for few-shot intent detection.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1209–1218.
  * Nguyen et al. (2023b) Hoang Nguyen, Chenwei Zhang, Tao Zhang, Eugene Rohrbaugh, and Philip Yu. 2023b.  Enhancing cross-lingual transfer via phonemic transcription integration.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 9163–9175, Toronto, Canada. Association for Computational Linguistics.
  * Qin et al. (2022) Libo Qin, Qiguang Chen, Tianbao Xie, Qixin Li, Jian-Guang Lou, Wanxiang Che, and Min-Yen Kan. 2022.  Gl-clef: A global–local contrastive learning framework for cross-lingual spoken language understanding.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2677–2686.
  * Shou et al. (2022) Ziyi Shou, Yuxin Jiang, and Fangzhen Lin. 2022.  AMR-DA: Data augmentation by Abstract Meaning Representation.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 3082–3098, Dublin, Ireland. Association for Computational Linguistics.
  * Wang et al. (2023) Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. 2023.  Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2609–2634, Toronto, Canada. Association for Computational Linguistics.
  * Wang et al. (2022) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022.  Self-consistency improves chain of thought reasoning in language models.  _arXiv preprint arXiv:2203.11171_.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022.  Chain of thought prompting elicits reasoning in large language models.  _arXiv preprint arXiv:2201.11903_.
  * Weld et al. (2022) Henry Weld, Xiaoqi Huang, Siqu Long, Josiah Poon, and Soyeon Caren Han. 2022.  A survey of joint intent detection and slot filling models in natural language understanding.  _ACM Computing Surveys_ , 55(8):1–38.
  * Wu et al. (2020) Di Wu, Liang Ding, Fan Lu, and Jian Xie. 2020.  SlotRefine: A fast non-autoregressive model for joint intent detection and slot filling.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1932–1937, Online. Association for Computational Linguistics.
  * Xia et al. (2020) Congying Xia, Chenwei Zhang, Hoang Nguyen, Jiawei Zhang, and Philip Yu. 2020.  Cg-bert: Conditional text generation with bert for generalized few-shot intent detection.  _arXiv preprint arXiv:2004.01881_.
  * Xie et al. (2022) Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I. Wang, Victor Zhong, Bailin Wang, Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragomir Radev, Caiming Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2022.  UnifiedSKG: Unifying and multi-tasking structured knowledge grounding with text-to-text language models.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 602–631, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Xu et al. (2021) Zenan Xu, Daya Guo, Duyu Tang, Qinliang Su, Linjun Shou, Ming Gong, Wanjun Zhong, Xiaojun Quan, Daxin Jiang, and Nan Duan. 2021.  Syntax-enhanced pre-trained model.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 5412–5422.
  * Yao et al. (2022) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022.  React: Synergizing reasoning and acting in language models.  _arXiv preprint arXiv:2210.03629_.
  * Zhang et al. (2019) Chenwei Zhang, Yaliang Li, Nan Du, Wei Fan, and S Yu Philip. 2019.  Joint slot filling and intent detection via capsule neural networks.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5259–5267.
  * Zhang et al. (2020) Zhuosheng Zhang, Yuwei Wu, Junru Zhou, Sufeng Duan, Hai Zhao, and Rui Wang. 2020.  Sg-net: Syntax-guided machine reading comprehension.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 9636–9643.
  * Zhao et al. (2023a) Ruochen Zhao, Xingxuan Li, Shafiq Joty, Chengwei Qin, and Lidong Bing. 2023a.  Verify-and-edit: A knowledge-enhanced chain-of-thought framework.  _arXiv preprint arXiv:2305.03268_.
  * Zhao et al. (2023b) Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023b.  A survey of large language models.  _arXiv preprint arXiv:2303.18223_.
  * Zhou et al. (2022) Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, et al. 2022.  Least-to-most prompting enables complex reasoning in large language models.  In _The Eleventh International Conference on Learning Representations_.
  * Zhou et al. (2023) Han Zhou, Ignacio Iacobacci, and Pasquale Minervini. 2023.  XQA-DST: Multi-domain and multi-lingual dialogue state tracking.  In _Findings of the Association for Computational Linguistics: EACL 2023_ , pages 999–1009, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Zhuang et al. (2021) Liu Zhuang, Lin Wayne, Shi Ya, and Zhao Jun. 2021.  A robustly optimized bert pre-training approach with post-training.  In _Proceedings of the 20th Chinese National Conference on Computational Linguistics_ , pages 1218–1227.
  * Zoph et al. (2022) Barret Zoph, Colin Raffel, Dale Schuurmans, Dani Yogatama, Denny Zhou, Don Metzler, Ed H. Chi, Jason Wei, Jeff Dean, Liam B. Fedus, Maarten Paul Bosma, Oriol Vinyals, Percy Liang, Sebastian Borgeaud, Tatsunori B. Hashimoto, and Yi Tay. 2022.  Emergent abilities of large language models.  _TMLR_.

##  Appendix A Connections between Semantic Parsing and NLU Tasks via Logic
Form

Logic Form not only captures the coarse-grained intent labels and fine-grained
slot labels of the utterances but also encapsulates the implicit connections
between slots and intents.

As observed in Table 7, Logic Form is constructed as the flattened
representation of the dependency structure between intents and slot sequences.
Semantic Frame constructed as intent type(s) followed by a sequence of slot
types can be directly extracted from the Logic Form. In addition, via the
Logic Form, the coarse-grained intent label CREATE_REMINDER, fine-grained slot
TODO, DATE_TIME labels together the respective slot values (message Mike, at
7pm tonight) can all be extracted and converted to appropriate format (i.e.
BIO format as the traditional sequence labeling ground truths Zhang et al.
(2019)). Therefore, Logic Form can be considered the unified label format to
bridge the gap between Semantic Parsing Li et al. (2021); Xie et al. (2022)
and traditional Intent Detection and Slot Filling tasks in NLU systems Xia et
al. (2020); Nguyen et al. (2020); Casanueva et al. (2022).

##  Appendix B Dataset Details

Table 5: Details of MTOP and MASSIVE datasets

Dataset | MTOP | MASSIVE
---|---|---
# Domains | 11 | 18
# Train Domains | 8 | 14
# Test Domains | 3 | 4
# Intents | 117 | 60
# Slots | 78 | 55
Sentence Length | 6.14 $\pm$ 2.30 | 6.34 $\pm$ 2.94
# Slots per sample | 1.87 $\pm$ 0.81 | 0.73 $\pm$ 0.63

We provide the details of MTOP and MASSIVE datasets in Table 5. As compared to
MASSIVE, MTOP dataset not only contains more slot types and intent types but
also tends to cover more slot types per sample in the Logic Form. This
challenging characteristic explains the consistent lower performance across
all methods on MTOP when compared to MASSIVE as observed in Section 5.

##  Appendix C Implementation Details

As the proposed step-by-step reasoning can be applied to any LLMs, our
proposed method is LLM-agnostic which is empirically studied in Appendix F.
For simplicity and consistency, in our main empirical study, we leverage
gpt-3.5-turbo from OpenAI as the base LLM model. Following Wang et al. (2022),
we set the decoding temperature T=0.7 and number of outputs n=10.

As domain names provide essential clues for language models in multi-domain
settings for multi-grained NLU tasks Zhou et al. (2023), to safeguard the
fairness in baseline comparisons, we consistently include the domain name in
the input prompts for all baselines unless stated otherwise. Specifically, the
only exception is presented in Table 4 for CoF-CoT(CoF order)-Conditioning.

For few-shot (i.e. k-shot) learning settings, we randomly sample k examples
and manually prepare the necessary labels for different baseline variants. We
experiment with k=5 in our empirical study.

#### Domains of Demonstration Samples

To replicate a more realistic scenario where the domains of k-shot
demonstration samples are generally unknown, we assume that k-shot
demonstration samples come from different domains from the test samples. The
relaxation of constraints on the assumption regarding the domain similarity
between demonstration samples and test samples allows for broader applications
and encourages LLMs to accumulate and extract the true semantic knowledge from
k-shot demonstrations and avoid overfitting any specific domains. For
completeness, we also conduct additional empirical studies to compare the FSL
performance of CoF-CoT under both scenarios: (1) k demonstration samples are
from the same domain as test samples, (2) k demonstrations are drawn from
different domains from the test samples. As observed in Table 6, additional
constraint of similar domains between k-shot demonstration samples and test
samples leads to improvements in the evaluation performance across NLU and
Semantic Parsing tasks. This might be intuitive since LLMs can extract domain-
relevant information from the given k domain-similar samples to assist with
inference process on test samples.

Table 6: FSL Results of CoF-CoT with k-shot demonstration samples selected
from different and similar domains in comparison with domains of test samples
on MTOP dataset.

Method | Assumption | Intent Acc | Slot F1 | Frame Acc | Exact Match
---|---|---|---|---|---
CoF-CoT | k domain-different samples | 61.50 $\pm$ 4.93 | 30.12 $\pm$ 3.93 | 15.00 $\pm$1.32 | 11.0 $\pm$ 1.61
CoF-CoT | k domain-similar samples | 70.00 $\pm$ 1.33 | 38.16 $\pm$ 5.42 | 20.50 $\pm$ 2.00 | 15.00 $\pm$ 1.00

##  Appendix D Prompt Design

Prompts for individual steps of our CoF-CoT are presented in Figure 3.
Additional output samples are also provided in Figure 4.

##  Appendix E Qualitative Case Study

We present additional Qualitative Case Study comparing the outputs between
different baseline methods and our proposed CoF-CoT in Figure 5.

As observed in Figure 5, our CoF-CoT provides the predictions closest to the
ground truth while other baselines struggle to (1) generate the correct intent
type (i.e. GET_DATE_TIME_EVENT intent type from Direct Prompt in comparison
with GET_EVENT intent from ground truth) (2) identify the correct slot values
(i.e. everything slot value generated from CoT), (3) generate the correct slot
type for the corresponding slot values. (i.e. EVENT_TYPE slot type for music
festivals slot values from Complex-CoT instead of CATEGORY_EVENT slot type).

Table 7: Sample utterance with its Logic Form under both Semantic Parsing and
NLU tasks’ metrics. // denotes the separation between tokens of the given
utterance.

| Metric | Granularity Level | Format | Ground Truth
---|---|---|---|---
Input Sentence | – | – | – | Set // up// a // reminder // to // message // mike // at // 7pm // tonight
Logic Form | – | – | – | [IN:CREATE_REMINDER [SL:TODO: message mike] [SL:DATE_TIME: at 7pm tonight]]
NLU Tasks | Intent Accuracy | Coarse-grained | Intent Label | IN:CREATE_REMINDER
Slot F1 | Fine-grained | BIO Slot Sequence | O // O // O // O // O // B-TODO // I-TODO // B-DATE_TIME // I-DATE_TIME // I-DATE_TIME
Semantic Parsing Tasks | Frame Accuracy | Both | Logic Form |  IN:CREATE_REMINDER-SL:TODO-SL:DATE_TIME
Exact Match | Both | Logic Form | [IN:CREATE_REMINDER [SL:TODO: message mike] [SL:DATE_TIME: at 7pm tonight]]

Figure 3: CoF-CoT Prompt Design Template. {.} denotes the placeholder
argument.  Figure 4: Sample output from our CoF-CoT. {.} denotes the
placeholder corresponding to template in Figure 3.  Figure 5: Qualitative Case
Study among baseline variants and the proposed CoF-CoT. Ground Truth is shown
in red.

##  Appendix F LLM-Agnostic Capability

Our proposed CoF-CoT is LLM-agnostic since the focus of the work is on the
prompt design, which can be applied to any LLMs. As most LLMs rely on the high
quality of the designed prompts, our proposed CoF-CoT prompt design can be
used as input to any LLMs for zero-shot and in-context learning settings. This
is also similarly observed in CoT Wei et al. (2022), SC-CoT Wang et al. (2022)
and other comparable CoT methods. For further clarification, we report
additional empirical results of our proposed CoF-CoT applied to both of the
backbone PaLM Chowdhery et al. (2022) and GPT3.5 LLMs on the MTOP dataset
under both ZSL and FSL settings in Table 8. As observed in Table 8, CoF-CoT
prompting consistently outperforms the two backbone LLMs across all NLU and
Semantic Parsing tasks, demonstrating both the effectiveness and LLM-agnostic
capability of our proposed CoF-CoT.

Table 8: Experimental results on MTOP dataset under zero-shot few-shot multi-
domain settings with different LLM backbone architectures (PaLM Chowdhery et
al. (2022) and GPT3.5).

MTOP
---
Model | Zero-shot | Few-shot
| NLU | Semantic Parsing | NLU | Semantic Parsing
| Intent Acc | Slot F1 | Frame Acc | Exact Match | Intent Acc | Slot F1 | Frame Acc | Exact Match
PaLM | 16.67 $\pm$ 2.52 | 7.24 $\pm$ 1.00 | 3.17 $\pm$ 0.76 | 1.17 $\pm$ 0.76 | 48.83 $\pm$ 4.54 | 14.24 $\pm$ 1.58 | 4.17 $\pm$ 2.02 | 2.33 $\pm$ 0.76
PaLM + CoF-CoT | 42.33 $\pm$ 3.33 | 13.73$\pm$ 2.88 | 4.01 $\pm$ 0.15 | 3.50 $\pm$ 1.32 | 57.17 $\pm$ 3.79 | 21.47 $\pm$ 3.79 | 10.33 $\pm$ 3.06 | 6.67 $\pm$ 2.75
GPT3.5 | 31.50 $\pm$ 1.80 | 21.84 $\pm$ 2.83 | 8.33 $\pm$ 1.44 | 6.00 $\pm$ 1.32 | 51.33 $\pm$ 3.40 | 28.35 $\pm$ 3.24 | 11.00 $\pm$ 1.80 | 8.33 $\pm$ 1.00
GPT3.5 + CoF-CoT | 57.67 $\pm$ 2.75 | 23.47 $\pm$ 4.09 | 14.33 $\pm$ 1.52 | 9.00 $\pm$ 1.00 | 61.50 $\pm$ 4.93 | 30.12 $\pm$ 3.93 | 15.00 $\pm$ 1.32 | 11.00 $\pm$ 1.61


**footnotetext: These authors contributed equally to this work

# An Empirical Study of Scaling Instruction-Tuned
Large Multimodal Models

Yadong Lu∗1, Chunyuan Li∗2, Haotian Liu3, Jianwei Yang2, Jianfeng Gao2, Yelong
Shen1
1Microsoft Azure AI 2Microsoft Research 3University of Wisconsin–Madison

###### Abstract

Visual instruction tuning has recently shown encouraging progress with open-
source large multimodal models (LMM) such as LLaVA and MiniGPT-4. However,
most existing studies of open-source LMM are performed using models with 13B
parameters or smaller. In this paper we present an empirical study of scaling
LLaVA up to 33B and 65B/70B, and share our findings from our explorations in
image resolution, data mixing and parameter-efficient training methods such as
LoRA/QLoRA. These are evaluated by their impact on the multi-modal and
language capabilities when completing real-world tasks in the wild. We find
that scaling LMM consistently enhances model performance and improves language
capabilities, and performance of LoRA/QLoRA tuning of LMM are comparable to
the performance of full-model fine-tuning. Additionally, the study highlights
the importance of higher image resolutions and mixing multimodal-language data
to improve LMM performance, and visual instruction tuning can sometimes
improve LMM’s pure language capability. We hope this study makes state-of-the-
art LMM research at a larger scale more accessible, thus helping establish
stronger baselines for future research. Code and checkpoints will be made
public.

##  1 Introduction

Recent studies on large multimodal models (LMM) [9, 10] have been focused on
the methods of visual instruction tuning [12]. The results are promising:
_e.g.,_ the open-source project Large Language and Vision Assistant (LLaVA)
shows that training a 7B large language model (LLM) with multimodal
instruction-following data for 3 hours on 8 A-100 GPUs leads to a LMM with
strong visual understanding and reasoning capabilities in the wild:
reproducing some of the most appealing examples of the proprietary OpenAI
multimodal GPT-4 model [14]. A similar idea is explored in its co-current work
MiniGPT-4 [20]. It has rapidly become a prominent research topic, spurring the
development of numerous new models, benchmarks, and applications [10].
However, the high compute cost has led most existing studies to utilize 7B and
13B LLMs. Thus, the impact of significantly scaling up the model size to
_e.g.,_ 33B and 65B remains unexplored.

This study aims to fill this gap by empirically investigating language models
of larger sizes for LMM, sharing insights of our scaling experiments and
establishing stronger baselines using larger-scale LLaVA for future research.
Specifically, we explore the impact of larger model sizes, model tuning and
data mixing methods on model performance, and present our findings and
recommendations. The scaling recipe leads to new state-of-the-art (SoTA)
performance on LLaVA-Bench [12] and MM-VET [19]. We hope that our findings and
larger LLaVA checkpoints would provide a reference for future research on
visual instruction tuning.

##  2 Experiment Setup

#### Model Checkpoints.

To study the impact of scaling up LLM on multimmodal capabilities, we increase
the language model size to 33B and 65B [15], in addition to the 7B and 13B
models used for existing LMM.

  * •

LLaVA-33B We employ the open source Vicuna-33B checkpoint
111https://huggingface.co/lmsys/vicuna-33b-v1.3 [16] to preform the two-stage
training. The training data is around 125K conversations collected from
ShareGPT.com.

  * •

LLaVA-65B Due to a lack of public 65B Vicuna checkpoint, we conduct our own
training of the Vicuna-65B model, utilizing ShareGPT data that we have
independently processed. This data contains 159M tokens used during training.
As a comparison, the reported number of tokens used in training Vicuna 33B is
370M 222https://github.com/lm-
sys/FastChat/blob/main/docs/vicuna_weights_version.md.

Once the instruction-tuned LLM is given, we follow [12] to perform the two-
stage LLaVA lightning training: $(i)$ Stage 1: Pre-training for Feature
Alignment. The linear projection layer is trained, which maps the visual
feature (the features before the last layer of the pre-trained image encoder)
to word embedding space of LLM. More specifcally, the projection dimension is
1024$\rightarrow$6656 for the 33B model and 1024$\rightarrow$8192 for the 65B
model, respectively. In this stage, we use the concept-balanced subset of
LAION-CC-SBU data with 558K samples. $(ii)$ Stage 2: Visual Instruction
Tuning. We use the LLaVA-80K multimodal instruct dataset for the fine-tuning
stage. Various training schedules are explored to enable the model to follow
the diverse instructions to complete tasks in the wild, as to be detailed
below.

#### Tuning Methods.

We explore both the trainable modules and training data mixing for efficient
and effective visual instruct tuning of large models.

  * •

Trainable modules. In addition to tuning the linear projection layer, two
schemes are considered to tune the LLM: $(i)$ Full-model fine-tuning of LLM
and $(ii)$ Parameter-efficient training methods. For the latter, LoRA [7] and
QLoRA [4] are employed to allow us to tune large models with limited compute
resource. This aims to gain an in-depth understanding of the trade-off between
the training cost and model performance.

  * •

Data mixing. Typically only the multimodal instruction data is used in
Stage-2. We further consider mixing the language-only instruct data ShareGPT
with the LLaVA-80K multimodal instruction data to gain an in-depth
understanding of the trade-off between models’ language and multimodal
capabilities.

#### Hyper-parameters.

In the training process of both stages, we utilize the DeepSpeed library
333https://github.com/microsoft/DeepSpeed and employ the ZeRO3 optimizer,
except for QLoRA runs we use ZeRO2. We use a maximum sequence length of 2048.
For Stage 1, we train both the 33B and 65B models with a learning rate of
$1\\!\times\\!10^{-4}$ with no weight decay, and a learning rate with linear
decay and linear warmup for 3% of training steps in total. For Stage 2, we use
a learning rate of $2\\!\times\\!10^{-5}$ in full fine-tuning to train 1 epoch
for all the models in full finetuning, and a learning rate of
$1\\!\times\\!10^{-4}$ for the LoRA/QLoRA runs. We conducted a set of
hyperparameter search and for LoRA runs, and found larger LoRA alpha or
equivalently larger learning rate was crucial to get the best performance.
Specifically, we use LoRA alpha equals 2 times the LoRA rank, and a learning
rate of $1\\!\times\\!10^{-4}$, which works the best for all the models. For
full fine-tuning, we use a total batch size of 512 on 4 A100 nodes, where each
of these nodes is equipped with 8 A100-80G GPUs. For LoRA/QLoRA runs, we use a
total batchsize of 64 on 1 A100 node for 33B model and 2 nodes for 65B model.

##  3 Results

We first compare our large checkpoints on two recent benchmarks which are
specifically designed for LMM, then report our findings in the course of
scaling up LLaVA models.

Models | Reasoning | Conversation | Detail | Overall
---|---|---|---|---
Bard-0718 | 78.7 | 83.7 | 69.7 | 77.8
Bing-Chat-0629 | 90.1 | 59.6 | 52.2 | 71.5
LLaVA-13B (beam=1) | 81.7 | 64.3 | 55.9 | 70.1
LLaVA-13B (beam=5) | 84.3 | 68.4 | 59.9 | 73.5
LLaVA-33B (beam=1) | 82.9 | 70.2 | 62.6 | 73.9
LLaVA-33B (beam=5) | 83.5 | 72.6 | 61.9 | 74.8
LLaVA-65B (beam=1) | 87.3 | 63.8 | 62.3 | 74.2
LLaVA-65B (beam=5) | 88.7 | 59.4 | 65.7 | 74.4
Table 1: The performance comparison on LLaVA-Bench. Beam search sizes at 1 and
5 are reported.

Model | Rec | OCR | Knowledge | Generation | Spatial | Math | Total
---|---|---|---|---|---|---|---
Results of various open-source LMM on reported in the MM-VET paper [19]
LLaMA-Adapter v2-7B [5] | 16.8 | 7.8 | 2.5 | 3.0 | 16.6 | 4.4 | 13.6±0.2
OpenFlamingo-9B [1, 2] | 24.6 | 14.4 | 13.0 | 12.3 | 18.0 | 15.0 | 21.8±0.1
MiniGPT-4-8B [20] | 27.4 | 15.0 | 12.8 | 13.9 | 20.3 | 7.7 | 22.1±0.1
BLIP-2-12B [11] | 27.5 | 11.1 | 11.8 | 7.0 | 16.2 | 5.8 | 22.4±0.2
LLaVA-7B [12] | 28.0 | 17.1 | 16.3 | 18.9 | 21.2 | 11.5 | 23.8±0.6
MiniGPT-4-14B [20] | 29.9 | 16.1 | 20.4 | 22.1 | 22.2 | 3.8 | 24.4±0.4
Otter-9B [8] | 28.4 | 16.4 | 19.4 | 20.7 | 19.3 | 15.0 | 24.6±0.2
InstructBLIP-14B [3] | 30.8 | 16.0 | 9.8 | 9.0 | 21.1 | 10.5 | 25.6±0.3
InstructBLIP-8B [3] | 32.4 | 14.6 | 16.5 | 18.2 | 18.6 | 7.7 | 26.2±0.2
LLaVA-13B [12] | 30.9 | 20.1 | 23.5 | 26.4 | 24.3 | 7.7 | 26.4±0.1
MM-ReAct-GPT-3.5 [18] | 24.2 | 31.5 | 21.5 | 20.7 | 32.3 | 26.2 | 27.9±0.1
LLaVA-7B (LLaMA-2) [12] | 32.9 | 20.1 | 19.0 | 20.1 | 25.7 | 5.2 | 28.1±0.4
LLaVA-13B (V1.3, 336px) [12] | 38.1 | 22.3 | 25.2 | 25.8 | 31.3 | 11.2 | 32.5±0.1
LLaVA-13B (LLaMA-2) [12] | 39.2 | 22.7 | 26.5 | 29.3 | 29.6 | 7.7 | 32.9±0.1
MM-ReAct-GPT-4 [18] | 33.1 | 65.7 | 29.0 | 35.0 | 56.8 | 69.2 | 44.6±0.2
Results with our own experiment runs
LLaVA-13B (LLaMA-2) | 38.4 | 21.0 | 26.3 | 28.8 | 28.0 | 7.7 | 32.6±0.1
LLaVA-33B | 38.5 | 25.0 | 26.2 | 28.2 | 29.2 | 7.7 | 32.9±0.3
LLaVA-33B (Data Mixing) | 37.7 | 27.1 | 26.2 | 28.6 | 28.1 | 11.5 | 34.1±0.3
LLaVA-65B | 39.2 | 28.2 | 26.2 | 28.3 | 33.0 | 15.0 | 35.5±0.3
LLaVA-65B (Data Mixing) | 41.8 | 27.9 | 30.4 | 32.3 | 30.5 | 7.3 | 36.4±0.2

Table 2: Performance of various open-source LMM on MM-VET. Note that MM-ReAct
is not an single multimodal model, it is a system built on chaining visual
tools via GPT-3.5 or GPT-4, which we append as a reference. Our experiment run
on LLaVA-13B (LLaMA-2) yields very similar score with the same checkpoint
reported in MM-VET paper, indicating that our evaluation pipelines are
consistent.

###  3.1 Comparisons on Benchmarks

#### LLaVA-Bench.

LLaVA-Bench (In-the-Wild)444https://github.com/haotian-
liu/LLaVA/blob/main/docs/LLaVA_Bench.md [12] is a diverse evaluation dataset
consisting of 24 images with 60 questions in total, including indoor and
outdoor scenes, memes, paintings, sketches. Each image is paired with a
manually-curated, detailed description and a set of properly-selected
questions related to open-ended visual chat scenarios. Each questions belongs
to one of three types of tasks: conversations that contain simple visual
recognition & QA questions, detailed descriptions that characterize the image
with a long paragraph, and a complex reasoning task that focuses on deducing
implications from an image. Language GPT-4 (gpt4-0314) is used to score to the
generated answers. The relative scores between the model output and gold
response are reported. We compare LLaVA against the commercial visual chat
systems including Microsoft BingChat555https://www.bing.com/chat and Google
Bard666https://bard.google.com/ on LLaVA-Bench [12].

The results are presented in Table 1. The 33B and 65B checkpoints outperform
the 13B LLaVA model and Bing Chat. Despite the fact that LLaVA-Bench is small
(thus the comparison might not be statistically significant), the results are
encouraging: compared to large LMM, small open-sourced LMM are far more cost-
effective to be deployed in real-world applications. With negligible increase
of inference latency, we can significantly improve the performance for all
model sizes by increasing the beam search size from 1 to 5. Our results show
that larger LLaVA models generally exhibit better performance in tasks
involving complex reasoning and generating detailed descriptions, which
requires strong language competencies from larger LLM. In addition, larger
LLaVA models obtain comparable results to BingChat in multi-turn, multi-modal
conversation tasks that require strong image understanding capability.

#### MM-VET.

MM-VET [19] is designed based on the assumption that the intriguing capability
of solving complicated tasks is often achieved by a generalist LMM which is
able to integrate a varity of vision-language (VL) capabilities. MM-Vet
contains 200 images and 218 questions (samples), aiming to evaluate6 core VL
capabilities (recognition, OCR, knowledge, language generation, spatial
awareness, and math) and their combinations. For evaluation, an LLM-based
evaluator (gpt4-0613) is used to score open-ended outputs of different forms.
In Table 2, we report the results on MM-VET. The performance is consistently
improved from 13B to 33B and 65B. The largest LLaVA model improves SoTA
performance among the end-to-end open-source LMM. The most significant
improvements are observed when evaluating the capabilities of knowledge and
generation, followed by recognition and OCR. The performance on spatial and
math remains comparable. The result reveals that the improved LLM capability
is instrumental in storing more knowledge in the weights and leading to a
stronger language responding capability.

###  3.2 Scaling up LLaVA

The experiments are conducted to answer three research questions.

#### \raisebox{-0.9pt}{1}⃝ Which scaling factor matters?

We study the relative contribution of three scaling-up factors to the
performance improvement of LLaVA. The results are summarized in Table 3 (a).

  * •

Model size. Increasing the model size consistently improves the overall
performance. We conjecture that larger data size is essential to train a
larger model. For example, if we only train on LLaVA-80K data, we see smaller
gain when model size becomes larger.

  * •

Image resolution. By fixing the CLIP ViT image encoder, we compare the
variants that are pre-trained to take image resolution $224\times 224$ and
$336\times 336$, and find that the higher resolution consistently yields 2-3
points improvement across all four LLM sizes.

  * •

Data mixing. Larger models tend to have higher capability of fitting the
instruction data. By mixing the language-only instruction data (ShareGPT) with
LLaVA-80K, we can improve model performance by 2 points, compared to training
on multimodal instruction data only.

In Table 3 (b), we present our result on MM-Bench [13], which contains a set
of 2,974 questions, which evaluate models’ reasoning skills of six categories.
The combination of the three factors improve the baseline LLaVA 7B model,
reported in [13].

Image Size | Data Mixing | 7B | 13B | 33B | 65B
---|---|---|---|---|---
224$\times$224 | ✗ | 63.6 | 67.1 | 69.3 | 70.3
336$\times$336 | ✗ | 65.9 | 70.1 | 72.0 | 72.3
336$\times$336 | ✓ | – | – | 73.9 | 74.2

(a) Performance scores on LLaVA-Bench.

Checkpoint | Image Size | Data Mixing | Overall | LR | AR | RR | FP-S | FP-C | CP
---|---|---|---|---|---|---|---|---|---
LLaVA-7B | 224$\times$224 | ✗ | 36.2 | 15.9 | 53.6 | 28.6 | 41.8 | 20.0 | 40.4
LLaVA-33B | 336$\times$336 | ✓ | 55.7 | 23.3 | 74.0 | 46.0 | 51.5 | 50.4 | 67.2
LLaVA-65B | 336$\times$336 | ✓ | 56.0 | 24.4 | 72.3 | 49.3 | 50.5 | 51.2 | 68.1

(b) Performance scores on MM-Bench. The skills to evaluate include logic
reasoning (LR), attribute reasoning (AR), relation reasoning (RR), fine-
grained single-instance perception (FP-S), fine-grained cross-instance
perception (FP-C), and coarse perception (CP).

Table 3: The performance to scale up model size, image resolution and data
mixing.

#### \raisebox{-0.9pt}{2}⃝ When should the parameter-efficient training method
be considered?

As model size increases, it becomes necessary to consider using tuning methods
that are more efficient than full-model fine-tuning. LoRA and QLoRA are well-
known parameter-efficient tuning methods. As shown in Table 4, we report
compute cost using GPU hours per node, because the unit can be equivalent to
the price $13.63/hour (ND A100 v4 series) on Azure
777https://azure.microsoft.com/en-us/pricing/details/machine-learning/. The
total cost can be estimated by multiplying the #hours and #epochs.

In Table 4(a), we train both the 33B and 65B model with LoRA rank 8 and 64 for
1 epoch on the LLaVA-80K instruction-tuning dataset. For models with 33B
parameters and above, as we increase the LoRA rank values, we notice an
increase in both performance and cost until full-model tuning reaches its
maximum performance for a specific model size. In the case of the 13B model,
we find that a rank of 64 can deliver comparable performance to full-model
tuning. The cost is more related to the total number of parameters than the
number of trainable parameters. The cost increase due to raising the LoRA rank
for a given model size is significantly smaller than the cost increase by
enlarging model sizes. For example, increasing the LoRA rank from 8 to 64
nearly matches the performance as LoRA fine-tuning a 65B model with same rank,
but only requires 50% of 65B model’s training cost. In practice we find that
tuning 33B model provide a good trade-off between cost and performance.

Different LoRA variations have similar performance, and QLoRA requires lower
GPU memory cost and running-time cost than LoRA. When large models (_e.g.,_
65B) are trained with DeepSpeed ZeRO2 mode, they can fit into GPU with QLoRA,
while yield the OOM issue with LoRA. In the experiments, we find that the
hyperparameters of LoRA have a large impact of performance:$(i)$ Large
learning rate and alpha value of LoRA improves the results significantly. For
example, With the same rank=64, we reduce the learning rate=$2\times 10^{-5}$
and alpha=16, the performance decrease from 71.8 to 65.5 on LLaVA-Bench.
$(ii)$ Under the same setting, large ranks leads to little improvement.
_e.g.,_ we increase the rank from 64 to 128 and 512, it improves from 65.5 to
66.1 and 68.1, respectively.

| 7B | 13B | 33B | 65B
---|---|---|---|---
LoRA Rank | Full | 64 | Full | 8 | 64-QLoRA | 64 | Full | 64 | Full
Performance $\uparrow$ | 65.9 | 70.1 | 70.1 | 70.3 | 71.6 | 71.8 | 72.0 | 72.2 | 72.3
Time (GPU Hours per node) $\downarrow$ | 1.3 | 2.1 | 2.3 | 4.62 | 4.68 | 4.79 | 5.80 | 9.17 | 13.50
# Trainable Parameters (B) $\downarrow$ | 7 | 0.26 | 13 | 0.06 | 0.49 | 0.49 | 33 | 0.81 | 65

Table 4: The trade-off between performance and compute cost among different
model sizes and traing methods on LLaVA-80K data. “Full” indicates the full-
model fine-tuning. “Time” is reported as the total GPU hours to finish 1 epoch
training (running time $\times$ #GPUs) divided by 8 (#GPUs per node). All
models are trained on LLaVA-80K data, results are obtained through averaging 3
repeated evaluation runs with same set up on LLaVA-Bench.

#### \raisebox{-0.9pt}{3}⃝ A LMM with strong capabilities in both language and
multimodal?

We expand our evaluation in two aspects: $(i)$ MM-VET is added to measure the
integrated multimodal capabilities of LMM; $(ii)$ The pure language ability of
LMM is measured using Vicuna-80 [16] and MMLU [6], where the former evaluates
the instruct-following ability in real-world language tasks, the latter
evaluates the multilingual multi-task language ability. The results are shown
in Table 5, where all models are full-model fine-tuned.

Compared to Vicuna which initializes the LLM weights of LLaVA, it is
surprising to observe that LLaVA, after being trained solely on multimodal
instruction data, exhibits a comparable language capability. Mixing language
instruction data can boost LLaVA’s multimodal ability, but not the language
ability. This is partially attributed to the inclusion of complex reasoning
questions, and long-form answers in LLaVA-Instruct-158K, which helps maintain
the language capabilities of LLaVA. We also train LLaVA-70B based on the
LLaMA-2-70B-Chat checkpoint [15], and find that mixed results on multimodal
and language abilities. Interestingly, we improve LLaMA-2-70B-Chat by 2.4
points on MMLU, yielding an overall MMLU score of 65.1, which is the best
performance for the 70B model size, according to [17] and the Chatbot Arena
Leaderboard 888https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard.
To the best of our knowledge, this is the first reported result which shows
visual instructing tuning improve language ability of large-scale LMM.

Model | Data Mix | Multimodal | Language
---|---|---|---
LLaVA-Bench | MM-VET | Vicuna-80 | MMLU
Vicuna-13B | - | - | - | 79.9 | 55.8
LLaVA-13B | ✗ | 70.1 | 32.5 | 79.6 | 55.0
Vicuna-33B | - | - | - | 85.6 | 59.0
LLaVA-33B | ✗ | 72.0 | 32.9 | 85.3 | 56.1
LLaVA-33B | ✓ | 73.9 | 34.1 | 80.3 | 58.6
Vicuna-65B | - | - | - | 83.2 | 62.5
LLaVA-65B | ✗ | 72.3 | 35.5 | 84.5 | 62.6
LLaVA-65B | ✓ | 74.2 | 36.4 | 82.6 | 62.2
LLaMA-2-70B-Chat | - | - | - | 84.7 | 63.1
LLaVA-70B | ✓ | 69.8 | 35.4 | 81.3 | 65.1
Table 5: Performance on both multimodal and language capabilities.

##  4 Conclusions and Limitations

We present an empirical study of scaling the language model size for LMM. Our
main findings are: $(i)$ Scaling LMM consistently enhances model performance,
resulting in significant improvements in language capabilities, primarily due
to the increased LLM model size. We leave it to future work how to scale the
vision encoder to enhance the visual capabilities and improve model
performance on vision recognition and understanding tasks. $(ii)$ Parameter-
efficient methods such as LoRA/QLoRA are viable solutions to finetune large-
scale LLMs for a good performance-cost trade-off in some real-world settings
with limited GPU memory. We observe that LoRA/QLoRA’s performance are
comparable to that of fine-tuning the full model, establishing their
effectiveness through significant cost reduction in both model training and
serving. $(iii)$ Our study of training data curation reveals that properly
selecting image resolutions and mixing multimodal-language data for model
training can significantly improve the performance of the resultant LMM. We
also show for the first time that visual instruction tuning can improve LMM’s
language capability. Note that the training datasets used in this study is
small. So, our findings are still preliminary. In future work, we will
experiment using much larger datasets to investigate in detail whether and how
different methods of training data selection and mixing can improve the
quality of much larger LMM.

## References

  * [1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  Advances in Neural Information Processing Systems, 35:23716–23736, 2022.
  * [2] Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al.  Openflamingo: An open-source framework for training large autoregressive vision-language models.  arXiv preprint arXiv:2308.01390, 2023.
  * [3] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi.  Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023.
  * [4] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer.  Qlora: Efficient finetuning of quantized llms.  arXiv preprint arXiv:2305.14314, 2023.
  * [5] Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al.  Llama-adapter v2: Parameter-efficient visual instruction model.  arXiv preprint arXiv:2304.15010, 2023.
  * [6] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.  Measuring massive multitask language understanding.  arXiv preprint arXiv:2009.03300, 2020.
  * [7] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  arXiv preprint arXiv:2106.09685, 2021.
  * [8] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu.  Otter: A multi-modal model with in-context instruction tuning.  arXiv preprint arXiv:2305.03726, 2023.
  * [9] Chunyuan Li.  Large multimodal models: Notes on CVPR 2023 tutorial.  arXiv preprint arXiv:2306.14895, 2023.
  * [10] Chunyuan Li, Zhe Gan, Zhengyuan Yang, Jianwei Yang, Linjie Li, Lijuan Wang, and Jianfeng Gao.  Multimodal foundation models: From specialists to general-purpose assistants.  arXiv preprint, 2023.
  * [11] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  arXiv preprint arXiv:2301.12597, 2023.
  * [12] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning, 2023.
  * [13] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al.  Mmbench: Is your multi-modal model an all-around player?  arXiv preprint arXiv:2307.06281, 2023.
  * [14] OpenAI.  Gpt-4 technical report, 2023.
  * [15] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  arXiv preprint arXiv:2307.09288, 2023.
  * [16] Vicuna.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.  https://vicuna.lmsys.org/, 2023.
  * [17] Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al.  How far can camels go? exploring the state of instruction tuning on open resources.  arXiv preprint arXiv:2306.04751, 2023.
  * [18] Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang.  Mm-react: Prompting chatgpt for multimodal reasoning and action, 2023\.
  * [19] Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang.  Mm-vet: Evaluating large multimodal models for integrated capabilities.  arXiv preprint arXiv:2308.02490, 2023.
  * [20] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.  Minigpt-4: Enhancing vision-language understanding with advanced large language models.  arXiv preprint arXiv:2304.10592, 2023.