# SwitchGPT: Adapting Large Language Models for Non-Text Outputs

Xinyu Wang1, Bohan Zhuang2, Qi Wu1 Correspondence: Prof. Qi Wu
<EMAIL_ADDRESS>
###### Abstract

Large Language Models (LLMs), primarily trained on text-based datasets,
exhibit exceptional proficiencies in understanding and executing complex
linguistic instructions via text outputs. However, they falter when requests
to generate non-text ones. Concurrently, modality conversion models, such as
text-to-image, despite generating high-quality images, suffer from a lack of
extensive textual pretraining. As a result, these models are only capable of
accommodating specific image descriptions rather than comprehending more
complex instructions. To bridge this gap, we propose a novel approach,
SwitchGPT, from a modality conversion perspective that evolves a text-based
LLM into a multi-modal one. We specifically employ a minimal dataset to
instruct LLMs to recognize the intended output modality as directed by the
instructions. Consequently, the adapted LLM can effectively summon various
off-the-shelf modality conversion models from the model zoos to generate non-
text responses. This circumvents the necessity for complicated pretraining
that typically requires immense quantities of paired multi-modal data, while
simultaneously inheriting the extensive knowledge of LLMs and the ability of
high-quality generative models. To evaluate and compare the adapted multi-
modal LLM with its traditional counterparts, we have constructed a multi-modal
instruction benchmark that solicits diverse modality outputs. The experiment
results reveal that, with minimal training, LLMs can be conveniently adapted
to comprehend requests for non-text responses, thus achieving higher
flexibility in multi-modal scenarios. Code and data will be made available at
https://github.com/xinke-wang/SwitchGPT.

## Introduction

The emergence of Large Language Models (LLMs) (Brown et al. 2020; Zhang et al.
2022; Touvron et al. 2023a; Chowdhery et al. 2022) that are capable of
understanding and executing complex linguistic tasks has been a remarkable
advancement in recent years. Their prowess in comprehending, processing, and
generating text-based responses has paved the way for groundbreaking
applications. This includes fields such as natural language understanding
(Wang et al. 2019), automated question-answering systems (Lin, Hilton, and
Evans 2022), and conversational AI assistants (OpenAI 2023). Predominantly,
these models are trained on extensive crowdsourced text-based datasets. Such
datasets encompass a myriad of topics and languages, capturing the vast
expanse of human knowledge (Raffel et al. 2020; Radford et al. 2019).
Nonetheless, due to their text-centric training, traditional LLMs primarily
operate with textual inputs and outputs, resulting in unsatisfactory responses
when tasked with generating non-textual outputs (see Figure 1). Despite not
being directly exposed to non-textual data such as images and speech during
their training, recent research (Li et al. 2023a; Wang et al. 2023; Li et al.
2023c; Su et al. 2023; Zhu et al. 2023) has indicated that LLMs possess a
profound potential to understand such non-textual data. This revelation opens
the door to evolving pure text-based LLMs into a multi-modal paradigm. For
example, Mini-GPT4 (Zhu et al. 2023) trains a linear layer that connects
BLIP-2 (Li et al. 2023a) with Vicuna (Chiang et al. 2023), demonstrating the
possibility that LLMs can understand image inputs. However, studies on
enabling LLMs to produce non-textual outputs remain relatively limited,
restricting the LLM’s interactive capabilities in multimodal scenarios.

Figure 1: Given an instruction expecting a non-text response, text-based LLMs
like ChatGPT (OpenAI 2023) are constrained to providing text responses, while
popular text-to-image models such as Stable Diffusion (Rombach et al. 2022)
generate imagery based on direct description. In contrast, our proposed
SwitchGPT comprehensively interprets the underlying intent of the instruction,
accurately producing a more appropriate response.

Unlike traditional LLMs, which are primarily designed for unimodal
interaction, specifically text-to-text communication, modality conversion
models are adept at handling data across different modalities. For example,
image captioning (Vinyals et al. 2015; Hossain et al. 2019) exemplifies the
route of image$\rightarrow$text, whereas text-conditioned image generation
(Goodfellow et al. 2014; Ramesh et al. 2021) illustrates the transition from
text$\rightarrow$image. These models signal a significant advancement in
producing realistic samples across diverse data types. Yet, their training
chiefly relies on paired data, such as image-text pairs. The available volume
of such paired data is substantially smaller than single modality data, with
comparisons often being in the ballpark of hundreds of billion tokens (pure
text) v.s. mere hundreds of million pairs (image-text). As a result, modal
conversion models often lack the richness of knowledge and depth of
understanding exhibited by LLMs. This limitation also means they usually
struggle with comprehending and executing more complex instructions that rely
on acquired knowledge and common sense. (see Figure 1).

Given the aforementioned strengths and limitations of both LLMs and modality
conversion models, an intriguing proposition emerges: Can we amalgamate the
profound knowledge and understanding capabilities of LLMs with the modality
conversion models? Hence, the integrated models could potentially interpret
more intricate instructions and deliver outputs in various modalities. A
natural idea that emerges from this conundrum is to position the LLM as a
coordinator to orchestrate and utilize modality conversion models. Some recent
works (Schick et al. 2023; Shen et al. 2023; Lian et al. 2023) have
demonstrated the immense potential of leveraging LLMs as a controller,
showcasing their superior orchestration abilities to plan and execute fine-
grained instructions with external tools. For instance, HuggingGPT (Shen et
al. 2023) devised a workflow where ChatGPT is used as a controller to invoke
HuggingFace’s open-sourced models, thereby accomplishing sophisticated AI
tasks. However, it suffers from a few shortcomings. Inefficiency: HuggingGPT
operates as an online model, heavily leaning on OpenAI’s ChatGPT API. This
dependency necessitates frequent invocations of ChatGPT for functions such as
Task Planning and Model Selection, substantially inflating both the
operational cost and latency. Instability: HuggingGPT uses ChatGPT as a black
box without any tuning, and the LLM’s outputs are uncontrolled and do not
always return the desired results, leading to exceptions in the workflow.
Another example is LLM-Grounded Diffusion (Lian et al. 2023), which harnesses
the reasoning capabilities of LLMs to produce scene layouts in the form of
bounding boxes based on the input instructions. As a result, it offers more
meticulous control over the positioning and layout of objects with the
generated image. However, the inflexibility of such methods confines them to a
singular modality conversion pathway, and they often falter when interpreting
indirect requests. The above shortcomings highlight the challenges in current
integration efforts and the full potential of combining LLMs with modality
conversion models.

In this paper, we endeavor to address the above challenges and present an
instruction-tuned LLM to unlock its ability to generate non-text outputs, with
the help of several modality conversion models. The primary contributions of
this paper are threefold:

  * •

We present a Modality-aligned Instruction Tuning that efficiently enables LLMs
to discern the intended output modality as dictated by the instructions. Thus,
LLMs are empowered to summon appropriate modality conversion models for non-
text responses while fully retaining their original reasoning capabilities.
With this technique, any LLM can be easily adapted to produce non-text outputs
with minimal training.

  * •

We introduce a new evaluation set that includes thousands of instructions
targeting text, image, and audio outputs, to assess LLMs’ abilities in
handling multi-modal output requests, enabling a better understanding of
performance variations across different LLMs in multi-modal scenarios.

  * •

We conduct comprehensive experiments to validate our approach against state-
of-the-art open-source LLMs as well as OpenAI’s ChatGPT API in multi-modal
scenarios. The experimental results indicate that our models tuned with
modality-aligned instructions are able to retain their original reasoning
abilities while consistently generating accurate and appropriate non-text
outputs.

## Related Work

LLM as Controller. The AI community has witnessed a transformative surge in
Natural Language Processing advancements over recent years, largely driven by
the emergence of LLMs, such as GPT (Brown et al. 2020), OPT (Zhang et al.
2022), PaLM (Chowdhery et al. 2022), Bloom (Scao et al. 2022), and LLaMa
(Touvron et al. 2023a). The potential of LLMs extends beyond their immediate
function of text generation. With vast knowledge bases and intricate reasoning
capabilities, LLMs closely emulate human-like comprehension. With these
capabilities, a promising direction has emerged, which treats the LLM as a
controller. Instead of using LLMs merely as standalone entities, they can
serve as a coordinator between external tools (Shen et al. 2023; Schick et al.
2023; Li et al. 2023b; Qin et al. 2023) or even manipulate robotic systems
(Driess et al. 2023; Mai et al. 2023). For example, HuggingGPT (Shen et al.
2023) employs ChatGPT as a controller to manage a list of open-sourced models
in HuggingFace’s Hub to solve AI tasks. However, these methods often treat
LLMs as a black box, utilizing custom prompts for specific invocation rules,
leading to notably unstable outputs. Contrary to solely depending on pre-
trained LLMs for generating control commands, our method fine-tunes the LLM to
yield structured responses, thereby ensuring more consistent and reliable
outcomes.

Figure 2: Comparison of responses to the user’s instruction by different
models. While traditional text-to-image models like Stable Diffusion often
generate images based on superficial keywords, they might miss the underlying
intent of the instruction. Hugging-GPT can produce unstable results, for
instance, it outputs three responses in this case. LLM-Grounded Diffusion,
though adept at controlling the layout, still falls short in grasping the
deeper nuances behind the user’s request, as evidenced by the unwarranted
inclusion of a ‘book’ in its image. In contrast, our proposed approach not
only captures the true essence of the instruction but also aligns it with the
desired input for the modality conversion model, resulting in a more faithful
visual representation. (The figure is best viewed zoomed in.)

LLM Finetuning. Given that LLMs typically possess tens to hundreds of billions
of parameters and necessitate training on vast datasets, the barriers to their
training and fine-tuning are significantly raised, both in terms of
computational resources and data collection. Recently, many efforts have
delved into more resource-efficient methods of LLM finetuning. From a data
perspective, instruction tuning (Wang et al. 2022; Liu et al. 2023; Taori et
al. 2023) leverages high-performing, large-scale LLMs such as GPT-3.5 or GPT-4
to generate vast amounts of instructions and responses, which are then used to
fine-tune relatively smaller LLMs or even those large models themselves. From
the view of model parameters, Parameter Efficient Fine-Tuning (PEFT) explores
the use of post-training quantization (Dettmers et al. 2022; Frantar et al.
2023) or freezing LLM parameters to train an adapter (Hu et al. 2022; Dettmers
et al. 2023), with the goal of reducing computational overhead. Based on these
techniques, low-cost, customizable training of LLMs has become feasible, for
example, Alpaca (Taori et al. 2023) and Vicuna (Chiang et al. 2023)
significantly improve the performance of Llama (Touvron et al. 2023a) by
conducting an instruction tuning. LLaVA (Liu et al. 2023) presents visual
instruction tuning, enabling LLMs to understand image contents by tuning on
language-image paired instructions. However, these efforts either focus on
enhancing the LLM’s reasoning performance or on its image$\rightarrow$text
understanding abilities. In contrast, our approach delves deeper into
understanding the intention behind input instructions, enabling the LLM to
decide the most fitting output modality. This unique capability is achieved
using our proposed modality-aligned instruction tuning, which not only retains
the text$\rightarrow$text reasoning prowess of the LLMs but also enables them
to produce non-text outputs, including text$\rightarrow$image and
text$\rightarrow$speech.

Figure 3: Left: Flowchart illustrating the process of generating modality-
aligned instructions. By embedding the text from modality datasets into
prompts, our method ensures alignment between ChatGPT-generated instructions
and the modality conversion models. Right: Pipeline of instruction tuning and
inference. During training, the parameters of the LLM are frozen, and an
adapter is trained on the generated modality-aligned instructions dataset. At
the inference stage, the structured response of LLM is parsed and used to
select the appropriate modality conversion route to produce the final outputs.

Modality Conversion Models. Multi-modal applications have been a long-standing
research topic in the AI community. The unique capability of these models to
comprehend and translate between different data representations facilitates
conversions between various modalities, such as image captioning (Vinyals et
al. 2015; Hossain et al. 2019) (image$\rightarrow$text), text-conditioned
image generation (Goodfellow et al. 2014; Ramesh et al. 2021)
(text$\rightarrow$image), speech recognition (Ao et al. 2022)
(audio$\rightarrow$text), or even multiple modality conversion
(text/image$\rightarrow$text/image) such as Uni-Diffuser (Bao et al. 2023). We
specifically focus on text$\rightarrow$image and text$\rightarrow$speech
conversion in this paper. Given a text description, a text$\rightarrow$image
model aims at synthesizing an image that accurately depicts the content
described in the text. Early methods were mostly based on GAN (Xu et al. 2018;
Goodfellow et al. 2014) and VQ-VAE (Ramesh et al. 2021), while recently, due
to their stability and improved generation quality, diffusion-based (Rombach
et al. 2022) approaches have become increasingly popular. For
text$\rightarrow$speech, the goal is to transform a piece of text content to
corresponding audio. While these models can convert text into other
modalities, they struggle to understand complex instructions, making it
difficult to apply them to advanced interactive features, such as AI
assistants. To tackle these challenges, our method synergizes LLMs with
modality conversion models, thereby aligning the advanced reasoning abilities
inherent to LLMs with the conversion capabilities of multi-modal models.

## SwitchGPT

### Preliminary

Why multi-modal output matter? While the text remains the predominant medium
for human communication, images, and sounds often assume indispensable roles
in various contexts. For instance, a photograph capturing a golden sunset over
a tranquil beach can elicit emotions that mere words may find challenging to
express. Similarly, visually impaired individuals heavily depend on auditory
cues and descriptions to comprehend their surroundings. Therefore, the value
of non-text outputs cannot be understated.

However, there is still limited research on adapting LLMs for non-text
outputs. One primary reason for the limited research in this area is the
prohibitive cost associated with multi-modal LLM pre-training. A pioneering
work Emu (Sun et al. 2023) aims to bridge this gap by introducing a vision-
language pre-trained LLM. Emu is pre-trained on massive datasets, comprising
billions of image-text pairs and millions of video-subtitle pairs, using 128
A100 GPUs. This enables Emu to not only be capable of accepting image/text
inputs but also generate textual and visual outputs. Even though Emu has
achieved impressive performance on various tasks, its adaptability comes at a
high cost. For example, introducing a new modality such as audio necessitates
the collection of corresponding paired data and complete retraining.

Given the flourishing developments in recent years, the AI community has
amassed countless open-source models for diverse tasks. Integrating these
models into LLMs offers another solution for adapting them to non-text
outputs. This not only conserves computational resources and reduces carbon
emissions but also provides a cost-effective and smooth transition to newer
models without the need for extensive retraining. This paper falls into this
category. Existing methods such as HuggingGPT (Shen et al. 2023) can invoke
external models to generate non-text responses beyond the capabilities of
traditional LLMs like ChatGPT. However, its results are unstable. For example,
Figure 2 depicts that HuggingGPT may produce an audio response when an image
output is expected. Moreover, it inputs the full instruction as a prompt to
Stable Diffusion, leading to suboptimal results. The reasons leading to the
above issues can be summarized as follows:

  * •

Relying solely on designing prompts to utilize the zero-shot capability of LLM
for invoking external models can introduce ambiguity, leading to unstable
outputs.

  * •

The misalignment between LLMs outputs and external model inputs often results
in subpar performance.

### Modality-aligned Instruction Generation

To solve the aforementioned issues, we introduce Modality-aligned Instruction
Tuning (MaIT). The primary purpose of MaIT is twofold. First, MaIT aims to
cheaply tune the LLM $L$ to understand and interpret the expected output
modality $t$ from a given instruction $I$. Formally, this can be written as:

$L(I)=(r)\overset{\text{MaIT}}{\longrightarrow}L^{\prime}(I)=(r,t),$ (1)

where $L^{\prime}$ is the adapted LLM after instruction tuning, and $r$ is the
output response. For example, consider a simple instruction “What is the
answer to the following equation 1+1=?”. While $L$ might respond to ‘2’,
$L^{\prime}$ is expected to produce (‘2’, ‘text’), with ‘text’ being a flag
denoting the desired output modality. This modality type flag, $t$, informs
the LLM about when to invoke which modality conversion model. For instance, if
$t$ is ‘image’, the LLM knows to use text$\rightarrow$image conversion model
rather than simply giving the response $r$. However, recognizing when and
which model to invoke is not the sole requirement. It is equally crucial to
guide the LLM on how to use the modality conversion model. Without this
guidance, the LLM might misinterpret the instruction or use a misaligned
response as input when interfacing with the modality conversion model. This
can be seen in the failure example of HuggingGPT depicted in Figure 2, which
results in less-than-ideal outcomes. As such, a secondary goal of MaIT is to
ensure the LLM’s outputs align seamlessly with the inputs of the modality
conversion model. This is built upon the fact that for a single route of
modality conversion, such as text$\rightarrow$image, different models often
share almost the same training dataset. Therefore, instead of aligning the LLM
with a particular model, it is more effective to align it with the training
samples themselves. In other words, the objective is to minimize the
distribution gap between the LLM’s outputs and the textual descriptions in the
modal conversion task training dataset. We can rewrite Equation 1 to represent
this purpose as follows:

$L(I)=(r)\underset{\min\Delta(D_{r^{\prime}},D_{\text{text}\rightarrow
t})}{\overset{\text{MaIT}}{\longrightarrow}}L^{\prime}(I)=(r^{\prime},t),$ (2)

where $D_{r^{\prime}}$ and $D_{\text{text}\rightarrow t}$ respectively
represent the distribution of the output $r^{\prime}$ by the adapted
$L^{\prime}$, and the text description distribution of the training dataset
for the text$\rightarrow t$ task. To achieve this, we directly use the
training data from the modality conversion tasks, such as the caption of
images, to construct the response.

According to Equation 2, we are able to generate modality-aligned
instructions. Specifically, for the modality type $t$, we consider the three
most common modalities in this paper, _i.e._ , text, image, and speech.
Following the approaches of previous studies (Taori et al. 2023), we initiate
the process by designing seed instructions. Each record comprises three
components: the instruction, the anticipated output modality, and the
response. For example, a record might look like this: {“instruction”: “How do
you pronounce the name of the fast food brand with a yellow golden arch
logo?”, “response”: {“type”: “speech”, “McDonald’s”}}. Notably, with an
abundance of open-source text-to-text instruction datasets available, we
concentrated our efforts on devising seed instructions specifically for the
text$\rightarrow$image and text$\rightarrow$speech modalities. To generate
instructions in larger quantities, we employed OpenAI’s ChatGPT API. This
involved integrating descriptions from the modality conversion task’s training
data into the ChatGPT prompt to produce modality-aligned instructions. An
example of this process is illustrated in Figure 3. Using a picture caption
like “A black metal bicycle with a clock inside the front wheel”, we prompt
ChatGPT to formulate a suitable instruction that solicits the generation of
such an image. In this case, the ChatGPT comes up with an intriguing
instruction “Can you generate an image that represents the concepts of ‘time
travel’, using everyday objects such as a bicycle and a clock?”. This
showcases ChatGPT’s strong capability in producing diverse instructions.
Additionally, the original caption, combined with the specified modality type,
is employed to construct the ground-truth response. Specifically, we use the
LAION-aesthetic (Schuhmann et al. 2022) and LibriTTS (Zen et al. 2019) as
referenced modality datasets to sample image captions or speech contents. In
this manner, we synchronize the output of LLM with the input of the modality
conversion model. Importantly, since we exclusively used textual descriptions
of images for instruction creation, it is only necessary to conduct text-to-
text tuning on the LLM. This avoids potential computational overhead from
multi-modal fine-tuning. We will provide more details about the generated
instructions in the appendix.

Method | Modality Acc. (%) $\uparrow$ | Vision | Language | Speech
---|---|---|---|---
CLIP $\uparrow$ | FID $\downarrow$ | QA $\uparrow$ | BLEU $\uparrow$
OPT-2.7B (Zhang et al. 2022) | 68.4 | 16.6 | 131.3 | 0.53 | 0.07
OPT-6.7B (Zhang et al. 2022) | 16.2 | 6.4 | 277.3 | 0.66 | 0.00
Llama-7B (Touvron et al. 2023a) | 66.8 | 17.9 | 88.6 | 0.55 | 0.07
Llama2-7B (Touvron et al. 2023b) | 76.2 | 18.7 | 86.1 | 0.69 | 0.08
Alpaca-7B (Taori et al. 2023) | 72.3 | 17.2 | 94.7 | 0.58 | 0.07
Vicuna-7B (Chiang et al. 2023) | 73.1 | 17.0 | 95.0 | 0.59 | 0.07
GPT-3.5-turbo (OpenAI 2023) | 88.1 | 21.4 | 84.1 | 0.75 | 0.17
HuggingGPT (Shen et al. 2023) | 81.7 | 19.9 | 85.2 | 0.62 | 0.10
SwitchGPT-7B (Ours) | 86.9 | 22.6 | 82.4 | 0.67 | 0.16
Table 1: Comparison of the performance between the proposed SwitchGPT and
state-of-the-art models.

### Training and Inference

Once the modality-aligned instruction dataset has been generated, we integrate
it with the existing text-only instructions. This results in the training set
comprising three routes of instructions, _i.e._ , text$\rightarrow$text,
text$\rightarrow$image, and text$\rightarrow$speech. In line with previous
research, we maintain a comparable number of instructions for training,
totaling approximately 52k, with each route accounting for roughly one-third
of this total. While conducting instruction-tuning, it is essential to
preserve the original reasoning and generation capabilities of the LLM.
Therefore, for Equation 2, when $t$ is text, it is desired that the
distribution of responses $r$ and $r^{\prime}$ remain as similar as possible:

$\left\\{\begin{aligned} &\min\Delta(D_{r^{\prime}},D_{r})&&,t=\text{text}\\\
&\min\Delta(D_{r^{\prime}},D_{\text{text}\rightarrow t})&&,t\neq\text{text}\\\
\end{aligned}\right.$ (3)

This consistency is attained by incorporating the text$\rightarrow$text
instructions during the instruction-tuning phase. Furthermore, the parameters
of the LLM are kept constant. Only a Low-Rank adapter (LoRA) (Hu et al. 2022)
is fine-tuned, which not only safeguards the pre-trained weights but also
significantly reduces computational costs.

As shown in Figure 2, a challenge of the current method such as HuggingGPT
(Shen et al. 2023) is the unpredictable format of the output. Such variability
can lead to inconsistencies during subsequent parsing stages, often resulting
in exceptions. To address this issue, we encode all responses into a
structured JSON format, ensuring that the LLM produces outputs with a
consistent structure. Due to its extensive pre-training, LLM is already
familiar with the JSON formatting rules. Therefore, training with a modest
amount of data can effectively guide it to generate appropriately formatted
outputs. As shown in the right block of Figure 3, the instructional data
consists solely of text, so the LLM does not require interaction with any
modality conversion models during the training phase. When moving to the
inference stage, the LLM outputs a structured JSON format response for the
given instruction. This determines when to choose which modality conversion
model and how to use it. In this manner, all modality conversion models remain
static without any fine-tuning.

## Experiments

### Implementation

In implementing the proposed pipeline, we used the Llama-2 (Touvron et al.
2023b) as our foundation LLM. For modality conversion, StableDiffusion-v1-5
(Rombach et al. 2022) and SpeechT5 (Ao et al. 2022) models were employed. The
parameters of the LLM were loaded using Int8 (Dettmers et al. 2022) and
subsequently frozen. With the modality-aligned instructions generated by
GPT-3.5-turbo, we trained a LoRA adapter (Hu et al. 2022) upon the frozen LLM.
Our training was executed on four Nvidia A100 (40GB) GPUs (however, training
on a single GPU is possible), utilizing a per-device batch size of 4 and
gradient accumulation steps of 8. Optimization of the model was carried out
using the AdamW (Loshchilov and Hutter 2019) for 3 epochs, with a consistent
learning rate $3^{-4}$. The entire training procedure was efficiently
completed in approximately 3 hours.

### Validation Set and Metrics

Evaluating LLM performance objectively is crucial for contrasting different
methods. Current benchmarks, such as The Open LLM Leaderboard (Beeching et al.
2023), are primarily tailored to assess text$\rightarrow$text reasoning and
generation capabilities. In contrast, this paper focuses on enabling the LLM
to produce non-text outputs. Given this discrepancy, existing evaluation
standards are not suited for a comprehensive assessment of our model. To
address this gap, we introduce a new validation set (which will be released)
specifically designed to facilitate the assessment of multi-modal output LLMs.
This set contains 2,400 instructions, each demanding an output in one of three
modalities: text, image, or speech. For the text$\rightarrow$text
instructions, we sample questions from established benchmarks, including
Truthful-QA benchmark (Lin, Hilton, and Evans 2022) and MMLU (Hendrycks et al.
2021), applying a variant multiple-choice metric. The text$\rightarrow$image
instructions are bifurcated into two categories: 200 intricate instructions
crafted by humans, and 600 instructions chosen from proposals generated by
ChatGPT-4 using the COCO-caption (Chen et al. 2015) as a reference. The
performance of LLM on this task is evaluated using both CLIP (Radford et al.
2021) and FID (Heusel et al. 2017) scores. Regarding the
text$\rightarrow$speech tasks, a similar strategy as of the
text$\rightarrow$image is used to construct the instructions, while Libritts
(Zen et al. 2019) is employed as reference. As the text$\rightarrow$speech
model translates input text verbatim, the audio quality remains unaffected by
the LLM’s output. We thus employ the BLEU (Papineni et al. 2002) score to
gauge the similarity between the LLM’s output and the referenced contents.
Additionally, to ascertain if the model can discern the desired output
modality from the instruction, we evaluate its performance using
classification accuracy, and only those predictions that match the ground-
truth modality will be further assessed for their vision, language, or speech
scores. Notably, since the default output modality of LLM is text, these
outputs will not contribute to classification accuracy. We explain more
details about evaluation metrics in the appendix.

Method | CLIP | FID
---|---|---
Stable Diffusion v1-5 | 15.2 | 140.3
LLM-grounded Diffusion | 18.9 | 85.9
SwitchGPT-7B (Ours) | 22.6 | 82.4
Table 2: Comparison of the performance between Stable Diffusion v1-5, LLM-
grounded Diffusion, and our method for text$\rightarrow$image instructions.
Figure 4: Qualitative results. From left to right: images produced by Stable
Diffusion, LLM-grounded Diffusion, HuggingGPT, and the proposed SwitchGPT. The
right side of each image showcases the intermediate artifacts such as the
layout and prompts (Stable Diffusion uses instruction as prompt directly)
generated by the corresponding methods.

### Quantitative Results

To assess the efficacy of the proposed SwitchGPT, we compared them with a
range of state-of-the-art techniques. These include pretrained LLMs like OPT
(Zhang et al. 2022) and Llama (Touvron et al. 2023a, b); instruction-tuned
LLMs such as Alpaca (Taori et al. 2023) and Vicuna (Chiang et al. 2023); and
the commercial API, GPT-3.5-turbo. Additionally, we considered methods that
employ LLMs as controllers, such as HuggingGPT (Shen et al. 2023) and LLM-
grounded Diffusion (Lian et al. 2023). Given that many text-based LLMs are not
inherently designed to produce non-text responses, a few-shot assessment
approach was adopted. Specifically, four instructions and corresponding
responses are given as examples, prompting these methods to generate responses
to new inputs (please refer to supplementary for more details). Notably, all
of the LLMs share identical modality conversion models to produce non-text
responses.

Table 1 demonstrates the performance of the proposed method in comparison with
existing techniques. For modality accuracy, our method achieves comparable
performance with GPT-3.5-turbo and surpasses all other techniques. This
highlights its capability to understand the expected modality of the
instruction. Interestingly, the OPT-6.7B model lags considerably in accuracy
when compared to all other methods, even its own 2.7B counterpart. We observed
that it predominantly provides text output for most instructions. This
discrepancy might be attributed to its heightened sensitivity to specific
prompts and examples. The CLIP score indicates the congruence of the generated
images with their corresponding descriptions. As all LLM models utilize the
same text$\rightarrow$image foundation model, the score can be used to measure
the quality of the prompt generated by the LLM. the score serves as an
indicator of the prompt quality produced by the LLM. Our model consistently
excels in this metric, highlighting the effectiveness of the modality-
alignment instruction tuning. Regarding language scoring, our method achieves
an accuracy comparable to the original Llama2-7B. This suggests that the
foundational reasoning capability remains unaffected. Additionally, since
stable diffusion and LLM-grounded diffusion can only generate image results,
we compare their performance in Table 2.

### Qualitative Results

In Figure 4, we provide a qualitative comparison of various methods based on
instructions intended for image output generation. The results demonstrate
that our proposed method aligns well with modality conversion models, yielding
enhanced performance. Other methods, such as HuggingGPT, tend to use the
instruction itself as the prompt to input into the modality conversion model,
resulting in unsatisfactory outputs. Meanwhile, Figure 5 presents a demo (to
be released) showcasing an example conversation between our model and users on
the topic ‘The Statue of Liberty’. The interaction highlights our model’s
capability to discern the underlying intentions of the instructions, despite
these conversational inputs being quite different from the instructions used
during training. Consequently, it is capable of producing appropriate
responses involving different modalities. See more results in the appendix.

Figure 5: The proposed SwitchGPT is able to generate non-text responses to
fulfill the given instructions.

## Conclusion

In this paper, we have presented modality-aligned instruction tuning, a method
designed to adapt text-based LLMs for generating non-text responses. This is
the inaugural effort to align the outputs of LLMs and the inputs of modality
conversion models from a data-driven perspective. A key advantage is that it
allows LLMs to be trained without direct exposure to non-text modality data.
Instead of delving into the complex real of multi-modal pre-training, they
simply undergo a standard instruction tuning process, ensuring computational
efficiency. To objectively evaluate our method, we have introduced a
validation set comprising instructions that request text, image, and audio
outputs. Based on this dataset, we examined several state-of-the-art LLMs in a
few-shot setting and benchmarked our approach against them. We anticipate that
our proposed SwitchGPT will serve as a baseline for future studies in
multimodal output LLMs. In future research, one promising direction could be
to design adapters at the encoder stage that would enable the integration of
multi-modal inputs. By seamlessly connecting this feature with the method
proposed in this study, it could potentially give rise to a comprehensive Any-
to-Any LLM capable of handling a wider array of tasks and functionalities.

## References

  * Ao et al. (2022) Ao, J.; Wang, R.; Zhou, L.; Wang, C.; Ren, S.; Wu, Y.; Liu, S.; Ko, T.; Li, Q.; Zhang, Y.; et al. 2022.  Speecht5: Unified-modal encoder-decoder pre-training for spoken language processing.  In _Proc. Annu. Meet. Assoc. Comput. Linguist._
  * Bao et al. (2023) Bao, F.; Nie, S.; Xue, K.; Li, C.; Pu, S.; Wang, Y.; Yue, G.; Cao, Y.; Su, H.; and Zhu, J. 2023.  One transformer fits all distributions in multi-modal diffusion at scale.  _arXiv preprint arXiv:2303.06555_.
  * Beeching et al. (2023) Beeching, E.; Fourrier, C.; Habib, N.; Han, S.; Lambert, N.; Rajani, N.; Sanseviero, O.; Tunstall, L.; and Wolf, T. 2023.  Open LLM Leaderboard.  https://huggingface.co/spaces/HuggingFaceH4/open˙llm˙leaderboard.
  * Brown et al. (2020) Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020.  Language models are few-shot learners.  _Proc. Advances in Neural Inf. Process. Syst._ , 33: 1877–1901.
  * Chen et al. (2015) Chen, X.; Fang, H.; Lin, T.-Y.; Vedantam, R.; Gupta, S.; Dollár, P.; and Zitnick, C. L. 2015.  Microsoft coco captions: Data collection and evaluation server.  _arXiv preprint arXiv:1504.00325_.
  * Chiang et al. (2023) Chiang, W.-L.; Li, Z.; Lin, Z.; Sheng, Y.; Wu, Z.; Zhang, H.; Zheng, L.; Zhuang, S.; Zhuang, Y.; Gonzalez, J. E.; et al. 2023.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.  _See https://vicuna. lmsys. org_.
  * Chowdhery et al. (2022) Chowdhery, A.; Narang, S.; Devlin, J.; Bosma, M.; Mishra, G.; Roberts, A.; Barham, P.; Chung, H. W.; Sutton, C.; Gehrmann, S.; et al. 2022.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_.
  * Conover et al. (2023) Conover, M.; Hayes, M.; Mathur, A.; Meng, X.; Xie, J.; Wan, J.; Shah, S.; Ghodsi, A.; Wendell, P.; Zaharia, M.; et al. 2023.  Free dolly: Introducing the world’s first truly open instruction-tuned llm.
  * Dettmers et al. (2022) Dettmers, T.; Lewis, M.; Belkada, Y.; and Zettlemoyer, L. 2022.  Llm. int8 (): 8-bit matrix multiplication for transformers at scale.  _Proc. Advances in Neural Inf. Process. Syst._
  * Dettmers et al. (2023) Dettmers, T.; Pagnoni, A.; Holtzman, A.; and Zettlemoyer, L. 2023.  Qlora: Efficient finetuning of quantized llms.  _arXiv preprint arXiv:2305.14314_.
  * Driess et al. (2023) Driess, D.; Xia, F.; Sajjadi, M. S.; Lynch, C.; Chowdhery, A.; Ichter, B.; Wahid, A.; Tompson, J.; Vuong, Q.; Yu, T.; et al. 2023.  Palm-e: An embodied multimodal language model.  _arXiv preprint arXiv:2303.03378_.
  * Frantar et al. (2023) Frantar, E.; Ashkboos, S.; Hoefler, T.; and Alistarh, D. 2023.  Gptq: Accurate post-training quantization for generative pre-trained transformers.  In _Proc. Int. Conf. Learn. Representations_.
  * Goodfellow et al. (2014) Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; and Bengio, Y. 2014.  Generative adversarial nets.  In _Proc. Advances in Neural Inf. Process. Syst._ , volume 27.
  * Hendrycks et al. (2021) Hendrycks, D.; Burns, C.; Basart, S.; Zou, A.; Mazeika, M.; Song, D.; and Steinhardt, J. 2021.  Measuring massive multitask language understanding.  _Proc. Int. Conf. Learn. Representations_.
  * Heusel et al. (2017) Heusel, M.; Ramsauer, H.; Unterthiner, T.; Nessler, B.; and Hochreiter, S. 2017.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  _Proc. Advances in Neural Inf. Process. Syst._ , 30.
  * Hossain et al. (2019) Hossain, M. Z.; Sohel, F.; Shiratuddin, M. F.; and Laga, H. 2019.  A comprehensive survey of deep learning for image captioning.  _ACM Computing Surveys_ , 51(6): 1–36.
  * Hu et al. (2022) Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2022.  Lora: Low-rank adaptation of large language models.  In _Proc. Int. Conf. Learn. Representations_.
  * Li et al. (2023a) Li, J.; Li, D.; Savarese, S.; and Hoi, S. 2023a.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  In _Proc. Int. Conf. Mach. Learn._
  * Li et al. (2023b) Li, M.; Song, F.; Yu, B.; Yu, H.; Li, Z.; Huang, F.; and Li, Y. 2023b.  Api-bank: A benchmark for tool-augmented llms.  _arXiv preprint arXiv:2304.08244_.
  * Li et al. (2023c) Li, Y.; Hu, B.; Chen, X.; Ma, L.; and Zhang, M. 2023c.  LMEye: An Interactive Perception Network for Large Language Models.  _arXiv preprint arXiv:2305.03701_.
  * Lian et al. (2023) Lian, L.; Li, B.; Yala, A.; and Darrell, T. 2023.  LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models.  _arXiv preprint arXiv:2305.13655_.
  * Lin, Hilton, and Evans (2022) Lin, S.; Hilton, J.; and Evans, O. 2022.  Truthfulqa: Measuring how models mimic human falsehoods.  In _Proc. Annu. Meet. Assoc. Comput. Linguist._
  * Liu et al. (2023) Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023.  Visual instruction tuning.  _arXiv preprint arXiv:2304.08485_.
  * Loshchilov and Hutter (2019) Loshchilov, I.; and Hutter, F. 2019.  Decoupled weight decay regularization.  In _Proc. Int. Conf. Learn. Representations_.
  * Mai et al. (2023) Mai, J.; Chen, J.; Li, B.; Qian, G.; Elhoseiny, M.; and Ghanem, B. 2023.  LLM as A Robotic Brain: Unifying Egocentric Memory and Control.  _arXiv preprint arXiv:2304.09349_.
  * OpenAI (2023) OpenAI. 2023.  ChatGPT.  https://openai.com/blog/chatgpt.
  * Papineni et al. (2002) Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W.-J. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proc. Annu. Meet. Assoc. Comput. Linguist._ , 311–318.
  * Qin et al. (2023) Qin, Y.; Liang, S.; Ye, Y.; Zhu, K.; Yan, L.; Lu, Y.; Lin, Y.; Cong, X.; Tang, X.; Qian, B.; et al. 2023.  ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs.  _arXiv preprint arXiv:2307.16789_.
  * Radford et al. (2021) Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021.  Learning transferable visual models from natural language supervision.  In _Proc. Int. Conf. Mach. Learn._ , 8748–8763. PMLR.
  * Radford et al. (2019) Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; Sutskever, I.; et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8): 9.
  * Raffel et al. (2020) Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, P. J. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21(1): 5485–5551.
  * Ramesh et al. (2021) Ramesh, A.; Pavlov, M.; Goh, G.; Gray, S.; Voss, C.; Radford, A.; Chen, M.; and Sutskever, I. 2021.  Zero-shot text-to-image generation.  In _Proc. Int. Conf. Mach. Learn._ , 8821–8831. PMLR.
  * Rombach et al. (2022) Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; and Ommer, B. 2022.  High-resolution image synthesis with latent diffusion models.  In _Proc. IEEE Conf. Comp. Vis. Patt. Recogn._ , 10684–10695.
  * Scao et al. (2022) Scao, T. L.; Fan, A.; Akiki, C.; Pavlick, E.; Ilić, S.; Hesslow, D.; Castagné, R.; Luccioni, A. S.; Yvon, F.; Gallé, M.; et al. 2022.  Bloom: A 176b-parameter open-access multilingual language model.  _arXiv preprint arXiv:2211.05100_.
  * Schick et al. (2023) Schick, T.; Dwivedi-Yu, J.; Dessì, R.; Raileanu, R.; Lomeli, M.; Zettlemoyer, L.; Cancedda, N.; and Scialom, T. 2023.  Toolformer: Language models can teach themselves to use tools.  _arXiv preprint arXiv:2302.04761_.
  * Schuhmann et al. (2022) Schuhmann, C.; Beaumont, R.; Vencu, R.; Gordon, C.; Wightman, R.; Cherti, M.; Coombes, T.; Katta, A.; Mullis, C.; Wortsman, M.; et al. 2022.  Laion-5b: An open large-scale dataset for training next generation image-text models.  _Proc. Advances in Neural Inf. Process. Syst._ , 35: 25278–25294.
  * Shen et al. (2023) Shen, Y.; Song, K.; Tan, X.; Li, D.; Lu, W.; and Zhuang, Y. 2023.  Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface.  _arXiv preprint arXiv:2303.17580_.
  * Su et al. (2023) Su, Y.; Lan, T.; Li, H.; Xu, J.; Wang, Y.; and Cai, D. 2023.  Pandagpt: One model to instruction-follow them all.  _arXiv preprint arXiv:2305.16355_.
  * Sun et al. (2023) Sun, Q.; Yu, Q.; Cui, Y.; Zhang, F.; Zhang, X.; Wang, Y.; Gao, H.; Liu, J.; Huang, T.; and Wang, X. 2023.  Generative pretraining in multimodality.  _arXiv preprint arXiv:2307.05222_.
  * Taori et al. (2023) Taori, R.; Gulrajani, I.; Zhang, T.; Dubois, Y.; Li, X.; Guestrin, C.; Liang, P.; and Hashimoto, T. B. 2023.  Stanford Alpaca: An Instruction-following LLaMA model.  https://github.com/tatsu-lab/stanford˙alpaca.
  * Touvron et al. (2023a) Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozière, B.; Goyal, N.; Hambro, E.; Azhar, F.; et al. 2023a.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Touvron et al. (2023b) Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; et al. 2023b.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Vinyals et al. (2015) Vinyals, O.; Toshev, A.; Bengio, S.; and Erhan, D. 2015.  Show and tell: A neural image caption generator.  In _Proc. IEEE Conf. Comp. Vis. Patt. Recogn._ , 3156–3164.
  * Wang et al. (2019) Wang, A.; Singh, A.; Michael, J.; Hill, F.; Levy, O.; and Bowman, S. R. 2019.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _Proc. Int. Conf. Learn. Representations_.
  * Wang et al. (2023) Wang, G.; Ge, Y.; Ding, X.; Kankanhalli, M.; and Shan, Y. 2023.  What Makes for Good Visual Tokenizers for Large Language Models?  _arXiv preprint arXiv:2305.12223_.
  * Wang et al. (2022) Wang, Y.; Kordi, Y.; Mishra, S.; Liu, A.; Smith, N. A.; Khashabi, D.; and Hajishirzi, H. 2022.  Self-instruct: Aligning language model with self generated instructions.  _arXiv preprint arXiv:2212.10560_.
  * Xu et al. (2018) Xu, T.; Zhang, P.; Huang, Q.; Zhang, H.; Gan, Z.; Huang, X.; and He, X. 2018.  Attngan: Fine-grained text to image generation with attentional generative adversarial networks.  In _Proc. IEEE Conf. Comp. Vis. Patt. Recogn._ , 1316–1324.
  * Zen et al. (2019) Zen, H.; Dang, V.; Clark, R.; Zhang, Y.; Weiss, R. J.; Jia, Y.; Chen, Z.; and Wu, Y. 2019.  Libritts: A corpus derived from librispeech for text-to-speech.  In _Proc. Interspeech_.
  * Zhang et al. (2022) Zhang, S.; Roller, S.; Goyal, N.; Artetxe, M.; Chen, M.; Chen, S.; Dewan, C.; Diab, M.; Li, X.; Lin, X. V.; et al. 2022.  Opt: Open pre-trained transformer language models.  _arXiv preprint arXiv:2205.01068_.
  * Zhu et al. (2023) Zhu, D.; Chen, J.; Shen, X.; Li, X.; and Elhoseiny, M. 2023.  Minigpt-4: Enhancing vision-language understanding with advanced large language models.  _arXiv preprint arXiv:2304.10592_.

## Appendix

##  Appendix A Instruction Generation

Here, we show more details of the generation of modality-aligned instructions.

### Seed Instruction and Generation Prompts

We have manually crafted 100 seed instructions, of which 50 are intended for
image output and 50 for audio output. Listing 1 and Listing 2 show some
examples of image and audio seeds, respectively.

Listing 1: Examples of Image Seed Instruction

⬇

1{

2 "instruction": "Show me a good design for web UI.",

3 "output":

4 {

5 "type": "image",

6 "response": "A master web UI design concept sketch featuring clean layout,
intuitive navigation bar, and coherent color scheme."

7 }

8},

9{

10 "instruction": "Showcasing the city a hundred years from now.",

11 "output":

12 {

13 "type": "image",

14 "response": "A futuristic cityscape with towering skyscrapers and flying
vehicles."

15 }

16},

17{

18 "instruction": "Turn this sentence into an image: a photo of an astronaut
riding a horse on Mars.",

19 "output":

20 {

21 "type": "image",

22 "response": "A photo of an astronaut riding a horse on Mars"

23 }

24}

Listing 2: Examples of Speech Seed Instruction

⬇

1{

2 "instruction": "Generate a voice clip to introduce Wuhan.",

3 "output":

4 {

5 "type": "audio",

6 "response": "Wuhan, located in Central China, is known for its educational
institutions like Wuhan University and landmarks such as the Yellow Crane
Tower. Famous for its street food, the city embodies the spirit of modern
China with its rich history and vibrant culture."

7 }

8},

9{

10 "instruction": "How do you pronounce this name: John?",

11 "output":

12 {

13 "type": "audio",

14 "response": "John"

15 }

16},

17{

18 "instruction": "Perform a famous line from the movie 'The Godfather' using
voice.",

19 "output":

20 {

21 "type": "audio",

22 "response": "I'm going to make him an offer he can't refuse."

23 }

24},

25{

26 "instruction": "Read this tongue twister 'She sells sea shells by the
seashore. The shells she sells are surely seashells. So if she sells shells on
the seashore, I'm sure she sells seashore shells.'",

27 "output":

28 {

29 "type": "audio",

30 "response": "She sells sea shells by the seashore. The shells she sells are
surely seashells. So if she sells shells on the seashore, I'm sure she sells
seashore shells."

31 }

32}

After constructing seed instructions, we devised templates to prompt ChatGPT
to generate instructions in bulk.

You are asked to generate instructions based on given image descriptions. Here
are some requests:
1\. Each instruction should be short and concise, ideally 1 to 2 sentences
long in English. 2\. Each instruction should be based on the image
descriptions. But, try not to directly use the contents in descriptions to
form the instruction, and reduce the use of the words that already appeared in
descriptions. 3\. Do not generate a request that might lead to confusing
answers. For example, do not ask to generate a character without any further
information. 4\. The language of instructions should be diverse, try to use
different verbs for each instruction to maximize the diversity. Either an
imperative sentence or a question is permitted.
Here are some examples: // Each time we randomly insert 3 examples from seed
instructions here
[A photo of avocado] Instruction: Visualize the answer to this riddle: I’m a
fruit that’s green when I’m raw and black when I’m ripe. I’m buttery but I’m
not a dairy product. What am I? [A futuristic cityscape with towering
skyscrapers and flying vehicles.] Instruction: Showcasing the city a hundred
years from now. [Mythical creature with dragon’s body, phoenix’s wings, and
both covered in flames.] Instructions: Create a visual representation of a
mythical creature that combines elements of a dragon and a phoenix. The
following are some image descriptions, Please generate an appropriate
instruction for each of them: // Image captions are sampled from LAION-
aesthetics [A large passenger airplane flying over some palm trees] [A dog
watching a man cut a piece of food] …

The template provided above illustrates the prompt input for OpenAI’s GPT API
(OpenAI 2023), tailored for generating image instructions. To generate speech
instructions, we simply adjust some keywords, such as changing ‘image’ to
‘speech’, and use the text-to-speech training set as a reference. For each
time the API is utilized, three examples from the seed instructions are
embedded into the prompt, and 60 captions are sampled from the reference
dataset. Consequently, the API can deliver up to 60 instructions per call,
with a cost of $<$ USD$ 0.01 each time.

(a) Text-only Instructions (Taori et al. 2023)

(b) Modality-aligned Instructions (Ours)

Figure 6: Comparison of the distribution of some top root verbs and
corresponding nouns between the text-only instructions generated by Alpaca
(Taori et al. 2023) and ours modality-aligned instructions.

Based on the above method, we generated approximately 40k instructions with
expected image or audio outputs. Subsequently, we removed those that were
malformed and could not be parsed. Additionally, we eliminated instructions
that requested nonspeech audio, such as the sounds of a cat purring, thunder
clapping, or music, as well as those asking for sentences to be read in other
languages, such as Japanese or Arabic.

Figure 6 contrasts the distribution of root verbs and their corresponding
nouns between our modality-aligned instructions and a text-based instruction
dataset produced by Alpaca (Taori et al. 2023). As depicted, our instructions
contain a substantial number of non-text directives, like “Generate image”,
“Design logo”, and “Recite quote”, which were deliberately excluded by Alpaca.
We merge

##  Appendix B Evaluation

### Few-shot Inference for Text-based LLMs

In the main paper, we have explained that with proper prompts, the pre-trained
pure text-based LLMs are also capable of analyzing the intended output
modality of given instructions. Therefore, we designed the following template
prompt to evaluate these LLMs.

You are a helpful assistant. Given an instruction, your task is to generate an
appropriate JSON-formatted response. Here’s how you can do it:
1\. Determine the expected output modality of the given instruction. The
available options are ‘text’, ‘image’, and ‘audio’. 2\. Place the expected
modality in the ‘type’ field of your response. 3\. Proceed to fulfill the
instruction based on the determined modality: • If the desired modality is
‘text’ or ‘speech’, provide the response directly in the ‘response’ field. •
If the desired modality is ‘image’, generate a concise and brief prompt. This
prompt will be sent to a text-to-image model, which will then generate the
image that fulfills the request. Instruction: “Show me the architectural style
of the Baroque era.” Response: “‘type”: “image”, “response”: “Ornate, grand
Baroque-style palace with dramatic lighting and complex shapes.”’
Instruction: “Show me a tranquil Japanese Zen garden.” Response: ’“type”:
“image”, “response”: “A tranquil Japanese Zen garden with raked sand, smooth
rocks, moss, and a wooden bridge over a small pond.”’
Instruction: “How do you say ‘October 1, 2000’ in spoken English?” Response:
“‘type”: ”audio”, “response”: “October first, two thousand.”’
Instruction: “Create an audio clip with Easter as the theme.” Response:
“‘type”: “audio”, “response”: “Easter, a cherished holiday, is celebrated as a
time of renewal and rebirth. It brings joy with its symbols of spring, such as
eggs, bunnies, and flowers.”’
Instruction: “What are the three primary colors?” Response: “‘type”: “text”,
“response”: “The three primary colors are red, blue, and yellow.”’
Instruction: {instruction} Response:

During the inference stage, the ‘{instruction}’ will be replaced with actual
inputs. While this method can measure the performance of existing methods to a
certain extent, it’s important to note that the outputs can vary significantly
depending on the method employed. A clear drawback of such approaches is the
need to encode lengthy explanations and examples into the prompt each time the
LLM is tasked with generating a response. This can be inefficient; for
instance, the template prompt itself contains over 400 tokens, leading to
significant redundancy. To ensure a fair comparison, all text-based LLMs
undergo few-shot inference using the identical template.

### Evaluation Instructions

There is no off-the-shelf suite to evaluate multi-modality output LLMs. Given
that the training set consists of short instructions generated by GPT,
employing similar instructions as a validation set for assessing model
performance is inappropriate. Hence, rather than using short instructions, we
utilize lengthier, conversation-like inputs for validation. This approach
avoids potential overlap with the training set and better assesses the model’s
generalization ability. We manually gathered conversation segments as
validation inputs (200 for each modality, excluding text). These segments,
combined with our instruction generation pipeline and manually filtered,
yielded a total of 800 validation instructions per modality. For the text
modality, given the abundance of existing evaluation sets, we directly sampled
800 entries from datasets like TruthfulQA (Lin, Hilton, and Evans 2022) and
MMLU (Hendrycks et al. 2021). Listing 3 shows some examples from the
validation set.

Listing 3: Examples of Validation Instruction

⬇

1{

2"instruction": "I am confused by so many types of camelids. Could you show me
a photo of different types of these creatures? Thank you!",

3"output": {

4 "type": "image",

5 "response": "A photo compares a llama, an alpaca, a vicuna, and a Gunaco
side by side.",

6 "image_id": "24531"

7}},{

8"instruction": "Can you show me the famous Japanese painting which includes
wave and mountain fuji?",

9"output": {

10 "type": "image",

11 "response": "The Great Wave off Kanagawa.",

12 "image_id": "42153"

13}}

Figure 7: Qualitative Results (Best viewed zoomed in). Figure 8: Failure
Cases.

### Metrics

Modality Classification Accuracy. We employ classification accuracy as the
metric to evaluate whether a method can accurately interpret the desired
output modality specified by the given instructions. It can be simply
calculated as follows:

$A=\frac{n_{correct}}{n_{total}}$ (4)

In this equation, $n_{correct}$ represents the number of correctly classified
modalities, while $n_{total}$ denotes the total count of test instructions.

Language. For language evaluation, existing datasets resembling multiple-
choice formats already provide several correct and incorrect answers for each
instruction. We employ the BLEU score to match the responses generated by LLMs
against the reference answers. Subsequently, we select the response that is
most similar to determine the final accuracy.

Image. We utilize two widely accepted metrics to assess the quality of
generated images based on their instructions: the CLIP and FID scores. The
CLIP score measures the relevance between the generated image and the
corresponding ground-truth descriptions. For instance, Listing 3 presents
several records from the test set. Within each record, there’s a textual
description of the anticipated image in the ‘response’ field, which we employ
to compute the CLIP score. On the other hand, the FID score evaluates the
quality of image generation by calculating the Fréchet inception distance
between the generated images and their reference counterparts. The referenced
image is indicated by the ‘image_id’ field in the records.

Speech. All LLMs utilize the same speech generation model, resulting in nearly
the same audio quality. Therefore, our primary concern is whether the content
aligns with the instructions. As such, we approach the evaluation of this task
as a translation/generation task, employing the BLEU score to measure the
similarity between the response and the ground truth.

##  Appendix C Qualitative Results

In the main paper, we have presented some qualitative results. Here, we show
more comparisons between our methods and existing methods.

### More comparisons

Figure 7 displays additional examples from both existing methods and our
approach. Several interesting observations are as follows:

  * •

While Stable Diffusion (Rombach et al. 2022) was not specifically optimized
for processing extended conversation-like inputs, it remains adept at
capturing key terms within such instructions. Thus, if the anticipated output
is evident or directly associated with the input, Stable Diffusion can still
produce satisfactory results. For instance, in the last row of Figure 7, the
term ‘supernova explosions’ is present in the input instruction, prompting
Stable Diffusion to generate a commendable representation of this concept.
Similarly, in the second row of Figure 7, even though the instruction does not
explicitly mention the painting’s title — ‘The Great Wave off Kanagawa’ —
Stable Diffusion creates an almost perfect image (though the position and size
of the mountain are different from the referenced image) based on the given
input. This proficiency might be attributed to Stable Diffusion’s prior
exposure to similar images during its training. Phrases like ‘wave’, ‘Japanese
painting’, and ‘Mount Fuji’ are sufficient cues for it to recognize and
retrieve from its stored knowledge. In essence, for widely recognized
concepts, Stable Diffusion can produce commendable results with sparse
information, leveraging its extensive pre-training. However, lesser-known
concepts or requests that require stronger reasoning ability, might not always
yield promising results.

  * •

LLM-grounded Diffusion (Lian et al. 2023), leveraging the extremely strong
reasoning capabilities of ChatGPT-4, yields promising results. However, when
converted to ChatGPT-3.5, it fails to produce any results for a majority of
inputs. Notably, for concepts that are composed of multiple sub-items, the
LLM-grounded Diffusion creates layouts for each individual item. This approach
can inadvertently segment a cohesive concept into disjointed fragments. For
instance, as depicted in the second row of Figure 7, the LLM-grounded
Diffusion separated the painting into two distinct elements: a wave and Mount
Fuji. Consequently, the final output appears as if two unrelated images were
pieced together, leading to gaps and extraneous content in their
juxtaposition.

  * •

In the case of HuggingGPT (Shen et al. 2023), given its support for multi-step
tasks, it occasionally produces unnecessary and irrelevant modality outputs.
For instance, it might frequently provide an audio clip alongside the output
image. Additionally, HuggingGPT sometimes struggles to properly parse its own
output, leading to unexpected errors. As depicted in the third row of Figure
7, HuggingGPT failed thrice in executing the provided instruction. Another
limitation is its misalignment between its output and the input of the text-
to-image models. Consequently, it often returns a segment of the instruction
as a prompt for image generation. This results in its outputs being notably
similar to the original stable diffusion in many cases.

### Failure Cases

In Figure 8, we present several failure cases of our proposed model,
SwitchGPT.

In the first row, all methods fail to recognize the intended entity ”a
colorful unicorn” from the inputs. Instead, they default to using the
description “a colorful horse-like animal” provided in the given instruction,
leading to erroneous outputs.

The second row illustrates that when given an instruction requesting a
specific layout, such as “two cups of beer in the center”, our method
predominantly relies on the modality conversion models to interpret the
prompt, even though it offers a relatively accurate description. On the other
hand, LLM-grounded Diffusion excels in this instance, being optimized for such
scenarios. It directly generates a layout with bounding boxes for two beers,
yielding an optimal result. Interestingly, HuggingGPT segments the instruction
into two parts. It uses the first segment “I am designing a poster for a beer
festival” for a text$\rightarrow$image task and then employs the generated
image, along with the second part “two cups of beer in the center of the
poster”, as prompts for an image$\rightarrow$image task. The final output
successfully places “two cups of beer in the center.”

In the third row, both LLM-grounded Diffusion and SwitchGPT fail to produce an
image, albeit for distinct reasons. The former’s failure arises because
ChatGPT-4 returns an empty object list for this instruction, preventing LLM-
grounded Diffusion from creating any images. In contrast, our model struggles
to discern the intended modality of the given instruction, leading it to
produce a purely textual response.