# Improving Image Captioning by Leveraging Knowledge Graphs

Yimin Zhou, Yiwei Sun, Vasant Honavar
Artificial Intelligent Research Laboratory
The Pennsylvania State University
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

We explore the use of a knowledge graphs, that capture general or commonsense
knowledge, to augment the information extracted from images by the state-of-
the-art methods for image captioning. The results of our experiments, on
several benchmark data sets such as MS COCO, as measured by CIDEr-D, a
performance metric for image captioning, show that the variants of the state-
of-the-art methods for image captioning that make use of the information
extracted from knowledge graphs can substantially outperform those that rely
solely on the information extracted from images.

##  1 Introduction

Advances in digital technologies have made it possible to acquire and share
vast amounts of data of all kinds, including in particular, images. The
availability of such data, together with recent advances in machine learning,
has resulted in robust and practical machine learning based solutions to
object recognition, e.g., Inception[1], vgg16[2], ResNet[3].

Recent years have witnessed a growing interest in describing visual scenes, a
task that is remarkably easy for humans yet remains difficult for machines
[4]. Of particular interest in this context is the image captioning problem,
which requires analyzing the visual content of an image, and generating a
caption, i.e., a textual description that summarizes the most salient aspects
of the image. Just as question answering presents challenges beyond text
processing, image captioning presents several challenges beyond image
processing. Effective image captions need to provide information that is not
explicit in the image, e.g., ”People gathered to watch a volleyball match”
when describing a crowd seated around a volleyball court, even if the image
shows no players on the field (perhaps because the game is yet to begin), or
”An impressionist painting of a garden by Claude Monet”, even if the image
makes no explicit mention of Monet or impressionism. Generating such captions
calls for incorporating background knowledge with information that is
available in the image. However, existing methods for image captioning (See
[5] for a review) fail to take advantage of readily available general or
commonsense knowledge about the world, e.g., in the form of knowledge graphs.

Inspired by the success of information retrieval and question answering
systems that leverage background knowledge [6], we explore an approach to
image captioning that uses information encoded in knowledge graphs.
Specifically, we augment the neural image caption (NIC) method introduced in
[7, 8] where a convolutional neural network (CNN) [9] trained to encodes an
image into a fixed length vector space representation or embedding and uses
the embedding to specify the initial state of a recurrent neural network (RNN)
that is trained to produce sentences describing the image in two important
aspects: In addition to a CNN trained to generate vector space embedding of
image features, we use an object recognition module that given an image as
input, produces as output, a collection of terms that correspond to objects in
the scene We use an external knowledge graph, specifically, ConceptNet [10,
11], a labeled graph which connects words and phrases of natural language
connected by edges that denote commonsense relationships between them, to
infer a set of terms directly or indirectly related to the words that describe
the objects found in the scene by the object recognition module. Vector space
embeddings of the terms as well as the image features are then used to specify
the initial state of an LSTM-based RNN that is trained to produce the caption
for the input image. We call the resulting image captioning system ConceptNet
enhanced neural image captioning system (CNet-NIC). The results of our
experiments on the MS COCO captions benchmark dataset [12] show that CNet-NIC
is competitive with or outperforms the state-of-the-art image captioning
systems on several of the commonly used performance measures (BLEU [13] ,
METEOR[14], ROUGE-L[15], all of which are measures designed originally for
evaluating machine translation systems as opposed to image captioning
systems). More importantly, CNet-NIC substantially outperforming the competing
methods on CIDEr-D, a variant of the CIDEr [16], the only measure that is
designed explicitly for evaluating image captioning systems. Because CIDEr-D
measures the similarity of a candidate image caption to a collection of human
generated reference captions, our results suggest that the incorporation of
background knowledge from ConceptNet enables CNet-NIC to produce captions that
are more similar to those generated by humans than those produced by methods
that do not leverage such background knowledge.

The rest of the paper is organized as follows. Section 2 summarizes the
related work on image captioning that sets the stage for our work on CNet-NIC.
Section 3 the design and implementation of CNet-NIC. Section 4 describes our
experimental setup and the results of our experiments assessing the
performance of CNet-NIC on the MS COCO image captioning benchmark dataset
along with comparisons with the competing state-of-the-art methods using the
standard performance measures (BLEU@$N$ ($N\in{1,2,3,4}$), METEOR, ROUGE-L,
and CIDEr-D) as well as a qualitative analysis of a representative sample of
the captions produced by CNet-NIC. Section 5 concludes with a summary and an
outline of some directions for further research.

##  2 Related Work

Existing image captioning methods can be broadly grouped into the following
(not necessarily disjoint) categories: (i) Template-based methods e.g., [17,
18, 19, 20] which rely on (often hand-coded) templates. Such methods typically
detect the object types, their attributes, scene types (e.g., indoor versus
outdoor), etc., based on a set of visual features, and generate image captions
by populating a template with the information extracted from the image. (ii)
Retrieval-based methods which can be further subdivided into two groups:
(ii.a) Image similarity based methods e.g., [21, 18, 22, 23, 24] which
retrieve captioned images that are visually most similar to the target image
and transfer their captions to the target image; and (ii.b) Multimodal
similarity based methods that use features of images as well as the associated
captions to retrieve or synthesize the caption for the target image [25, 26,
27, 28, 29, 30, 7]; (iii) Embedding-based methods, including those that use
recurrent, convolutional, or deep neural networks [7, 31, 32, 8, 33, 34, 35,
36] that make use of the learned low-dimensional embeddings of images to train
caption generators.

However, none of the existing methods take advantage of the readily available
background knowledge about the world e.g., in the form of knowledge graphs.
Such background knowledge has been shown to be useful in a broad range of
applications ranging from information retrieval to question answering [6],
including most recently, visual question answering (VQA) from images [37]. We
hypothesize that such background knowledge can address an important drawback
of existing image captioning methods, by enriching captions with information
that is not explicit in the image.

Unlike the state-of-the-art image captioning systems, CNet-NIC is specifically
designed to take advantage of background knowledge to augment the information
extracted from the image (image features, objects) to improve machine-produced
captions or image descriptions. Unlike VQA [37], which uses a knowledge graph
to extract better image features and hence better answer questions about the
image, CNet-NIC first detects objects (not just image features) in the image
and uses the detected objects to identify related terms or concepts which are
then used to produce better image captions.

##  3 CNet-NIC: ConceptNet-Enhanced Neural Image Captioning

We proceed to describe our design for an image captioning system that takes
advantage of background knowledge in the form of a knowledge graph.

###  3.1 CNet-NIC Architecture

Fig. 1 shows a schematic of the CNet-NIC system. CNet-NIC uses YOLO9000[38], a
state-of-the-art general-purpose real-time object recognition module that is
trained to recognize 9000 object categories. YOLO9000 takes an image as input
and produces as output, a collection of terms that refer to objects in the
scene. CNet-NIC use an external knowledge graph, specifically, ConceptNet [10,
11], a labeled graph which connects words and phrases of natural language
connected by edges that denote commonsense relationships between them, to
infer two sets of terms related to the words that describe the objects found
in the scene by the object recognition module. The first set of terms are
retrieved based on the individual objects in the scene. The second set of
terms are retrieved based on the entire collection of objects in the scene.
The resulting terms are then provided to a pre-trained RNN to obtain the
corresponding vector space embedding of the terms. A CNN is used to obtain
vector space embedding of the image features. The two resulting vector space
embeddings are used to specify the initial state of an LSTM-based RNN which is
trained to produce the caption for the input image.

Figure 1: Our model architecture by inviting common sense from external
resources

We proceed to describe each key element of the CNet-NIC system in detail.

###  3.2 Improving Image Captioning by Incorporating Background Knowledge

To test our hypothesis, we use the ConceptNet[10, 11], a kind of knowledge
graph, specifically, one that connects words and phrases of natural language
connected by edges that denote commonsense relationships between them.
ConceptNet integrates information from resources provided by experts as well
as through crowd-sourcing. It encodes general knowledge that is of use in
natural language understanding, and has been shown to enrich the semantic
information associated with words, beyond that supplied by distributional
semantics [11].

###  3.3 Generating Semantic Representations from ConceptNet

ConceptNet can be used to learn word embeddings using a variant of
”retrofitting” [39]. Let $V=\left\\{w_{1},\dots,w_{n}\right\\}$ be a
vocabulary, i.e., the set of word types, and $\Omega$ be an ontology encoding
semantic relations between words in $V$. $\Omega$ is represented as an
undirected graph $(V,E)$ with one vertex for each word type and edges
$(w_{i},\;w_{j})\in E\subseteq V\times V$ indicating a semantic relationship
of interest.

Let $\hat{Q}$ be the collection of vectors $\hat{q_{i}}\in\mathbb{R}^{d}$ for
each $w_{i}\in V$ that is learned using a standard data-driven method where
$d$ is the length of word vectors. The objective is to learn the matrix
$Q=(q_{1},\dots,q_{n})$ such that the columns are both close to their
counterparts in $\hat{Q}$ and to adjacent vertices in $\Omega$ where closeness
is measured using an appropriate distance measure, e.g., the Euclidean
distance. This is achieved by minimizing the following objective function:

$\Psi\left(Q\right)=\sum_{i=1}^{n}\left[\alpha_{i}\left\|q_{i}-\hat{q_{i}}\right\|^{2}+\sum_{\left(i,j\right)\in
E}\beta_{ij}\left\|q_{i}-q_{j}\right\|^{2}\right]$ (1)

where $\alpha$ and $\beta$ are parameters that control the the relative
strengths of associations. The procedure is called retro-fitting because the
word vectors are first trained independent of the information in the semantic
lexicons and are then retro-fitted by optimizing the objective function
specified above. Because $\Psi$ is convex in $Q$, the solution of the
resulting optimization problem is straightforward. $Q$ can be initialized to
$\hat{Q}$ and iteratively updated using the following update equation:

$q_{i}=\frac{\sum_{j:\left(i,j\right)\in
E}\beta_{ij}q_{j}+\alpha_{i}\hat{q_{i}}}{\sum_{j:\left(i,j\right)\in
E}\beta_{ij}+\alpha_{i}}$ (2)

###  3.4 Simple Recurrent Neural Network Image Caption Generator

Figure 2: The architecture of the simple recurrent neural network image
caption generator

We use a simple recurrent neural network image caption generator based on LSTM
introduced in [7] where a CNN is used to extract image features; and vector
space embedding of the extracted features is used by an LSTM-based RNN to
generate the caption text. The architecture of this model is shown in Fig. 2.

Let $X$ be an input image and $S=\left(S_{0},\dots,S_{N}\right)$ the
corresponding caption sentence. Let

$x_{-1}\;=\;CNN\left(X\right)$ (3)

$x_{t}\;=\;W_{e}S_{t},\;\;\;t\in\left\\{0\;\dots\;N-1\right\\}$ (4)

where $S_{t}$ is the one-hot vector representation of the word with a size of
the dictionary, $S_{0}$ a special start word, and $S_{N}$ a special end word.

$p_{t+1}\;=\;LSTM\left(x_{t}\right),\;\;\;t\in\left\\{0\;\dots\;N-1\right\\}$
(5)

The loss function is given by:

$L\left(I,S\right)=-\sum_{t=1}^{N}\log p_{t}\left(S_{t}\right)$ (6)

The loss function is minimized with respect to the parameters of the LSTM, CNN
and $W_{e}$.

###  3.5 Identifying Semantically Related Words

Given set of input words $W=\\{w_{1}\cdots w_{n}\\}$, and their associated
weights $u_{1}\cdots u_{n}$ (e.g., based on their frequency distribution), a
target word $w$ can be scored based on its semantic relatedness to the input
words, as measured by the weighted distance (e.g., cosine distance) between
the semantic vector representation of the query word with each of the target
words. Let $s_{w}$ and $s_{w_{i}}$ denote the semantic vector representations
of words $w$ and $w_{i}$ ($i\in\\{1,\cdots n\\}$; and $d(a,b)$ denote the
(cosine) distance between vectors $a$ and $b$.

$score(W,w)=\frac{{\displaystyle\sum_{i=1}^{n}}u_{i}d(s_{w},s_{w_{i}})}{\sum_{i=1}^{n}u_{i}}$
(7)

Figure 3: An example of relevant words generated by ConceptNet

Fig. 3 Identifying Semantically Related Words. Blue rectangular nodes denote
the input words (concepts). Red ovals denote the words that are most closely
related to the input words, whereas the green ovals the next most closely
related, and light blue ovals the next most closely related.

###  3.6 CNet-NIC

Let $X$ be an input image, and $O$ a set of terms corresponding to the objects
detected in the image $I$ by the YOLO9000 object recognition system. Thus,
$O=YOLO\left(X\right)$. For each $o\in O$, let
$r_{o}=ConceptNet\left(o\right)$ be the set of terms related to $o$ in the
ConceptNet knowledge graph; and $R_{O}=ConceptNet\left(O\right)$ the set of
terms related to the entire set $O$ of terms referring to all of the objects
detected in the image $X$ by the YOLO9000 object recognition system. Let
$D=\bigcup_{o\in O}r_{o}\cup\left\\{o\right\\}$ denote the set of terms
directly related to individual objects in $X$. Loosely speaking, $R_{O}$
provides terms that are descriptive of the scene as a whole, whereas $I$
provides terms that are descriptive of some or all of the objects depicted in
the image. Thus, $I=R_{O}-D$ denote a set of terms that are indirectly related
to objects in $X$. Let $d=RNN_{D}\left(D\right)$ and $i=RNN_{I}\left(I\right)$
denote the vector space embeddings of $D$ and $I$ produced by the pre-trained
RNNs $RNN_{D}$ and $RNN_{I}$ respectively. Let $a=CNN\left(X\right)$ be an
embedding of the image features of $X$ produced by a pre-trained CNN. The
image captions are produced by an LSTM-based RNN whose state is initialized as
follows:

$x_{-1}\;=\;a\parallel d\parallel i$ (8)

where $\parallel$ denotes the concatenation operation.

$x_{t}\;=\;W_{e}S_{t},\;\;\;t\in\left\\{0\;\dots\;N-1\right\\}$ (9)

where $S_{t}$ denotes the one-hot vector representation of the word with a
size of the dictionary, and $S_{0}$ a special start word, and $S_{N}$ a
special end word.

$p_{t+1}\;=\;LSTM\left(x_{t}\right),\;\;\;t\in\left\\{0\;\dots\;N-1\right\\}$
(10)

The cost function is given by:

$C=-\sum_{t=1}^{N}\log
p_{t}\left(S_{t}\right)+\lambda_{\theta}\cdot\left\|\theta\right\|_{2}^{2}$
(11)

where $\theta$ represents the model parameters and
$\lambda_{\theta}\cdot\left\|\theta\right\|_{2}^{2}$ is a regularization term.

Figure 4: The pre-trained RNN network architecture

The only trainable parameters are within the LSTM and $W_{e}$. The pre-trained
RNN network is shown in Fig. 4. Let $r_{i}$ be the $i$th word embedding.

$x_{i}=W_{r}r_{i},\;\;\;i\in\\{i,\;\dots\;L-1\\}$ (12)

$a=CNN\left(X\right)$ (13)

$a$ represents the image attribute embedding from CNN.

$y_{-1}=concatenate\left(a,\;x_{L-1}\right)$ (14)

$a=CNN\left(X\right)$ (15)

$y_{t}=W_{e}S_{t},\;\;\;t\in\left\\{0,\;...\;N-1\right\\}$ (16)

$p_{t+1}\;=\;LSTM\left(x_{t}\right),\;\;\;t\in\left\\{0\;\dots\;N-1\right\\}$
(17)

The cost function to be minimized is given by:

$C=-\sum_{t=1}^{N}\log p_{t}\left(S_{t}\right)$ (18)

Note that the only parameters to be trained are those associated with LSTM1,
LSTM2 and $W_{e}$. The trained LSTM1 is used in the model in Fig. 3 to extract
intermediate state of the related terms.

Table 1: Performance of our proposed models and other state-of-the-art methods on MS COCO dataset, where B@$N$, M, R, and C are short for BLEU@$N$, METEOR, ROUGE-L, and CIDEr-D scores. Except CIDEr-D, all values are reported as percentage(%). Model | B@1 | B@2 | B@3 | B@4 | M | R | C
---|---|---|---|---|---|---|---
NIC[7] | $66.6$ | $45.1$ | $30.4$ | $20.3$ | - | - | -
LRCN[36] | $62.8$ | $44.2$ | $30.4$ | $21$ | - | - | -
Soft Attention[31] | $70.7$ | $49.2$ | $34.4$ | $24.3$ | $23.9$ | - | -
Hard Attention[31] | $71.8$ | $50.4$ | $35.7$ | $25$ | $23$ | - | -
ATT[8] | $70.9$ | $53.7$ | $40.2$ | 30.4 | $24.3$ | - | -
Sentence Condition[40] | $72$ | $54.6$ | $40.4$ | $29.8$ | $24.5$ | - | $95.9$
LSTM-A[41] | 73 | $\bm{56.5}$ | $\bm{42.9}$ | $\bm{32.5}$ | 25.1 | 53.8 | 98.6
CNet-NIC | $\bm{73.1}$ | 54.9 | 40.5 | $29.9$ | $\bm{25.6}$ | $\bm{53.9}$ | $\bm{107.2}$
Table 2: Performance of variants of CNet-NIC on MS COCO dataset, where B@$N$, M, R, and C are short for BLEU@$N$, METEOR, ROUGE-L, and CIDEr-D scores. Except CIDEr-D, all values are reported as percentage(%). Input of Model | B@1 | B@2 | B@3 | B@4 | M | R | C
---|---|---|---|---|---|---|---
none(only seqs input) | $48.4$ | $24.7$ | $10.2$ | $3.9$ | $11$ | $34.2$ | $8.6$
image embedding | $70.3$ | $52.9$ | $38.3$ | $27.5$ | $24.3$ | $51.8$ | $99.5$
detected objects and directly related terms | $63.3$ | $43.4$ | $29.1$ | $20$ | $19.8$ | $46.1$ | $74.3$
indirectly related terms | $47.6$ | $27$ | $15.7$ | $10.2$ | $13.7$ | $36.6$ | $31.8$
detected objects and directly related terms + image embedding | $70.9$ | $53.3$ | $38.7$ | $28$ | $24.8$ | $52.4$ | $103.2$
indirectly related terms + image embedding | $70.1$ | $52.8$ | $38.2$ | $27.7$ | $24.5$ | $52$ | $100.5$
detected objects and directly related terms + indirectly related terms + image embedding | $72.1$ | $54.2$ | $38.9$ | $28.5$ | $24.8$ | $52.9$ | $103.6$
detected objects and directly related terms + indirectly related terms + image embedding + fine tune CNN | $73.1$ | $54.7$ | $40.5$ | $29.9$ | $25.6$ | $53.9$ | $107.2$

##  4 Experiments

###  4.1 Data

We used the Microsoft COCO captioning data set(COCO)[12], the most widely used
image captioning benchmark data set in our evaluations. The data set includes
82,783 images for training and 40,504 images for validation. For each image,
the data set includes 5 or 6 descriptions or captions provided by human
annotators. In our experiments, from the training and validation set provided,
we used 117,211 images for training, 2,026 images for validation and 4050
images for testing.

###  4.2 Experimental Setup

Details of the experimental setup are summarized below:

  * •

Data Preprocessing: Following [28], we convert all of the image captions in
training set to lower case and discard rare words which occur less than 4
times, resulting in the final vocabulary with 11,519 unique words in COCO data
set. Each word in the sentence is represented as ”one-hot” vector.

  * •

Attribute Extraction: To extract the image attributes, we use the output of
inception v3[42] network image recognition model pre-trained on the
ILSVRC-2012-CLS[43] image classification data set.

  * •

Object Detection: We use the YOLO9000 object detection network (with $23$
layers) and $544\times 544$ resolution. YOLO9000 is able to detect 9419 object
classes.

  * •

Leveraging Background Knowledge: Because YOLO9000 object detection system is
inherently imperfect, in identifying related terms using ConceptNet, we limit
ourselves to only the objects detected with high confidence. Based on
preliminary experiments, we set $30\%$ as the detection threshold.

  * •

Training the model: Our model is implemented on the TensorFlow platform in
Python language. The size of LSTM for each embedding(attributes, related
terms) is set to $512$. Initial learning rate is set to $2.0$ with an
exponential decay schedule. Batch size is set to $32$. Along the training, the
learning rate is shrunk by $5$ for three or four times. The number of
iterations is set to 500,000.

  * •

Testing the model: Two approaches can be utilized for sentence generation
during the testing stage. One approach is to select the word with maximum
probability at each time step and set it as LSTM input for next time step
until the end sign word is emitted or the maximum length of sentence is
reached. Another approach is to conduct a beam search that selects the top-$k$
best sentences at each time step and use them as the candidates to generate
the top-$k$ best sentences at the next time step. We adopt the second approach
and set the beam size $k$ empirically to $3$.

  * •

Evaluation Metrics: To evaluate CNet-NIC, we use 4 metrics: BLEU@$N$[13],
METEOR[14], ROUGE-E[15], and CIDEr-D[16]. All the metrics are computed by
using the codes released by [44].

Table 3: Image example of model showing ”Common Sense” from external resource.

Image |  Detected |  Indirectly Related |  Sentences Generated by Model with Indirectly Related |  Sentences Generated by Model without Indirectly Related |  Standard Model
---|---|---|---|---|---
|  Winsor chair, deck chair, furnishing, pot |  item of furniture, upholstered, found in house, chairs |  0) a dining room with a table and chairs
1) a dining room with a table, chairs and a table
2) a dining room with a table and chairs and a fireplace |  0) a table with a vase of flowers on it
1) a dining room with a table and chairs
2) a table with a vase of flowers on it |  0) a table with a vase of flowers on it
1) a table with a vase of flowers on it
2) a table with a vase of flowers on it
|  fishmonger, cereal bowl, phial, banana, waiter |  food storage jar, canaree, storing food, fruit bowl, food can |  0) a chef preparing food in a kitchen on a counter
1) a chef preparing food in a kitchen on a table
2) a man in a kitchen preparing food for a customer |  0) a man and a woman preparing food in a kitchen
1) a man and a woman preparing food in a kitchen
2) a chef preparing food in a kitchen next to a woman |  0) a group of people in a kitchen preparing food
1) a group of people standing around a kitchen
preparing food 2) a group of people in a kitchen preparing food
|  straight chair, furnishing |  item of furniture, reupholstery, end table |  0) a kitchen filled with appliances and lots of clutter
1) a kitchen filled with appliances and lots of counter space
2) a kitchen with a table and chairs |  0) a kitchen with a stove a sink and a counter
1) a kitchen with a stove a sink and a window
2) a kitchen with a stove top oven next to a sink |  0) a kitchen with a stove a sink and a stove
1) a kitchen with a stove a sink and a refrigerator
2) a kitchen with a stove a sink and a counter
|  book(s), toilet seat |  bookrack, bookshelving, bookrest |  0) a bathroom with a toilet and a book shelf
1) a bathroom with a toilet and a book shelf
2) a bathroom with a toilet and a sink |  0) a white toilet sitting in a bathroom next to a wall
1) a white toilet sitting next to a book shelf
2) a white toilet sitting in a bathroom next to a shelf |  0) a kitchen with a stove a sink and a stove
1) a kitchen with a stove a sink and a refrigerator
2) a kitchen with a stove a sink and a counter
|  trolleybus(es), park bench, commuter |  tram stop, bus rapid transit |  0) a couple of buses that are sitting in the street
1) a couple of buses that are parked next to each other
2) a couple of buses driving down a street next to a tall building |  0) a double decker bus driving down a street
1) a double decker bus driving down the street
2) a double decker bus is driving down the street |  0) a double decker bus driving down a street
1) a double decker bus driving down a city street
2) a double decker bus driving down the street
|  Friesian(s), Brown Swiss, private, settler |  dairy farm, cows, feed lot |  0) a group of cows standing next to each other
1) a group of cows that are standing in the dirt
2) a group of cows standing in a barn |  0) a group of cows that are standing in the dirt
1) a group of cows that are standing in the grass
2) a group of cows that are standing in a pen |  0) a group of cows are standing in a pen
1) a group of cows standing in a pen
2) a group of cows are standing in a field
|  overnighter(s), pilgrim(s), square dancer, general, peddler, hand luggage, backpack |  wayfaring, day tripper, journeyer, excursionist, traveller |  0) a group of people standing around with luggage.
1) a group of people standing around a luggage carousel.
2) a group of standing around a baggage claim area. |  0) a group of people standing around a luggage carousel.
1) a group of people standing around a luggage cart.
2) a group of people standing next to a luggage cart. |  0) a group of people standing around a luggage carousel.
1) a group of people standing around a luggage cart.
2) a group of people standing next to each other.
|  horse wrangler(s), Tennessee walker, wild horse |  found on ranch |  0) a man standing next to a brown horse.
1) a man is standing next to a horse
2) a man standing next a brown horse in a stable. |  0) a couple of people standing next to a horse.
1) a woman standing next to a brown horse.
2) a couple of people standing next to a brown horse. |  0) a group of people standing next to a horse.
1) a group of people standing next to a brown horse.
2) a group of men standing next to a brown horse.

###  4.3 Performance Comparison

We compare the performance of CNet-NIC with that of several state-of-the-art
image captioning methods (as reported in the respective papers):

  * •

Neural Image Caption (NIC)[7], which uses a vector space representation of
image features produced by a CNN to initialize an LSTM-based RNN trained to
generate image captions from vector space representation of image features.

  * •

Hard and Soft Attention[31], which combines two attention-based image
captioning mechanisms under an encoder-decoder framework: a “soft”
deterministic attention mechanism trainable by standard back-propagation
methods and 2) a “hard” stochastic attention mechanism that is trained using
reinforcement learning.

  * •

LRCN[36] which combines CNN with LSTMs to perform visual recognition and image
captioning.

  * •

ATT[8] which combines top-down and bottom-up attention models to extract image
features that are used to train an RNN to produce image captions.

  * •

Sentence-Condition[40] which uses a text-conditional attention mechanism for
focusing the caption generator on specific image features that should inform
the caption given the already generated caption text.

  * •

LSTM-A[41] which extends the basic LSTM model with image attributes model by
rearranging image and attributes input in different positions and time to
boost the accuracy of image captioning.

Table 1 shows the performance of each method on MS COCO image captioning data
set. Bold represents the best in that metric and italic represents the second
best. Overall the performance of CNet-NIC is comparable to or better than all
other models on all measures, especially with respect to CIDEr-D, the only
measure that is explicitly designed for the purpose of evaluating image
captions.

###  4.4 CNet-NIC Ablation Study Results

We report results of an ablation study of CNet-NIC, where we examine the
relative contributions of the different components of the CNet-NIC
architecture.

From the results summarized in Table 2, we see that detected objects and
directly related terms contribute to greater improvements in performance as
compared to indirectly related terms. We conjecture that the detected objects
and directly related terms provide more information about the individual
objects in an image whereas the indirectly related terms provide information
about the scene as a whole. This perhaps explains why only adding indirectly
related terms to image embedding improves performance as measured by METEOR,
ROUGE-L and CIDEr-D, albeit at the cost of a slight decrease in BLEU. We
further note that the indirectly related terms contribute to increases in
CIDEr-D, even when no image features are available. Overall, we find that
CNet-NIC which combines the background knowledge (ConceptNet derived terms)
related to the detected objects and the scene in generating image captions
outperforms all other methods that do not make use of such background
knowledge.

###  4.5 Qualitative Analysis of Captions

Table 3 presents several representative examples of captions produced by CNet-
NIC. Here we take a qualitative look at the captions to explore the role
played by the commonsense or background knowledge provided by the ConceptNet
knowledge graph. In the first example, the ConceptNet derived terms such as
”upholstered”, ”found in house”, etc. appear to yield more accurate captions.
In the third example, the standard model and the model without indirectly
related terms completely ignore the large furniture such as tables and chairs
while the model that incorporates indirectly related terms such as ”item of
furniture”, ”reupholstery”, ”end table”, etc. leads to what appear to be
better captions. For the fourth example, the indirectly related terms appear
to yield a more accurate caption model, e.g., one that mentions the book rack.
For the sixth image listed, only the model with indirectly related terms as
”dairy farm”, ”feed lot”, etc. from the knowledge graph correctly recognizes
that the scene is occurring in a ”barn”. In the seventh example, the model
with indirectly related terms correctly deduces that most people in the image
are travelers and conclude that they are in a baggage claim area. These
examples offer further qualitative evidence that shows the utility and
effectiveness of background knowledge supplied by knowledge graphs to improve
the quality of image captions.

##  5 Summary and Discussion

The focus of this paper is on the image captioning problem, which requires
analyzing the visual content of an image, and generating a caption, i.e., a
textual description that summarizes the most salient aspects of the image.
Image captioning presents several challenges beyond those addressed by object
recognition, e.g., inferring information that is not explicitly depicted in
the image. However, existing methods for image captioning (See [5] for a
review) fail to take advantage of readily available general or commonsense
knowledge about the world.

In this paper, we have presented CNet-NIC, an approach to image captioning
that incorporates background knowledge available in the form of knowledge
graphs to augment the information extracted from images. The results of our
experiments on several benchmark data sets, such as MS COCO, show that the
variants of the state-of-the-art methods for image captioning that make use of
the information extracted from knowledge graphs, e.g., CNet-NIC, can
substantially outperform those that rely solely on the information extracted
from images, e.g., CNet.

Some promising directions for future work include: variants and extensions of
CNet-NIC, including those that substantially improve the quality of captions,
provide justifications for the captions that they produce, tailor captions for
visual question answering, tailoring captions to different audiences and
contexts, etc. by bringing to bear on such tasks, all available background
knowledge.

## Acknowledgements

This project was supported in part by the National Center for Advancing
Translational Sciences, National Institutes of Health through the grant UL1
TR000127 and TR002014, by the National Science Foundation, through the grants
1518732, 1640834, and 1636795, the Pennsylvania State University’s Institute
for Cyberscience and the Center for Big Data Analytics and Discovery
Informatics, the Edward Frymoyer Endowed Professorship in Information Sciences
and Technology at Pennsylvania State University and the Sudha Murty
Distinguished Visiting Chair in Neurocomputing and Data Science funded by the
Pratiksha Trust at the Indian Institute of Science [both held by Vasant
Honavar]. The content is solely the responsibility of the authors and does not
necessarily represent the official views of the sponsors.

## References

  * [1] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, A. Rabinovich _et al._ , “Going deeper with convolutions.” Cvpr, 2015.
  * [2] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _CoRR_ , vol. abs/1409.1556, 2014. [Online]. Available: http://arxiv.org/abs/1409.1556
  * [3] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [4] L. Fei-Fei, A. Iyer, C. Koch, and P. Perona, “What do we perceive in a glance of a real-world scene?” _Journal of vision_ , vol. 7, no. 1, pp. 10–10, 2007\.
  * [5] R. Bernardi, R. Cakici, D. Elliott, A. Erdem, E. Erdem, N. Ikizler-Cinbis, F. Keller, A. Muscat, and B. Plank, “Automatic description generation from images: A survey of models, datasets, and evaluation measures,” _Journal of Artificial Intelligence Research_ , vol. 55, pp. 409–442, 2016\.
  * [6] Q. Wang, Z. Mao, B. Wang, and L. Guo, “Knowledge graph embedding: A survey of approaches and applications,” _IEEE Transactions on Knowledge and Data Engineering_ , vol. 29, no. 12, pp. 2724–2743, 2017.
  * [7] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, “Show and tell: A neural image caption generator,” in _Computer Vision and Pattern Recognition (CVPR), 2015 IEEE Conference on_. IEEE, 2015, pp. 3156–3164.
  * [8] Q. You, H. Jin, Z. Wang, C. Fang, and J. Luo, “Image captioning with semantic attention,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 4651–4659.
  * [9] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in _Advances in neural information processing systems_ , 2012, pp. 1097–1105.
  * [10] H. Liu and P. Singh, “Conceptnet—a practical commonsense reasoning tool-kit,” _BT technology journal_ , vol. 22, no. 4, pp. 211–226, 2004.
  * [11] R. Speer, J. Chin, and C. Havasi, “Conceptnet 5.5: An open multilingual graph of general knowledge.” in _AAAI_ , 2017, pp. 4444–4451.
  * [12] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _European conference on computer vision_. Springer, 2014, pp. 740–755.
  * [13] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in _Proceedings of the 40th annual meeting on association for computational linguistics_. Association for Computational Linguistics, 2002, pp. 311–318.
  * [14] S. Banerjee and A. Lavie, “Meteor: An automatic metric for mt evaluation with improved correlation with human judgments,” in _Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization_ , 2005, pp. 65–72.
  * [15] C.-Y. Lin, “Rouge: A package for automatic evaluation of summaries,” _Text Summarization Branches Out_ , 2004.
  * [16] R. Vedantam, C. Lawrence Zitnick, and D. Parikh, “Cider: Consensus-based image description evaluation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 4566–4575.
  * [17] K. Barnard, P. Duygulu, D. Forsyth, N. d. Freitas, D. M. Blei, and M. I. Jordan, “Matching words and pictures,” _Journal of machine learning research_ , vol. 3, no. Feb, pp. 1107–1135, 2003.
  * [18] A. Gupta and P. Mannem, “From image annotation to image description,” in _International Conference on Neural Information Processing_. Springer, 2012, pp. 196–204.
  * [19] A. Farhadi, M. Hejrati, M. A. Sadeghi, P. Young, C. Rashtchian, J. Hockenmaier, and D. Forsyth, “Every picture tells a story: Generating sentences from images,” in _European conference on computer vision_. Springer, 2010, pp. 15–29.
  * [20] G. Kulkarni, V. Premraj, V. Ordonez, S. Dhar, S. Li, Y. Choi, A. C. Berg, and T. L. Berg, “Babytalk: Understanding and generating simple image descriptions,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 35, no. 12, pp. 2891–2903, 2013.
  * [21] V. Ordonez, G. Kulkarni, and T. L. Berg, “Im2text: Describing images using 1 million captioned photographs,” in _Advances in neural information processing systems_ , 2011, pp. 1143–1151.
  * [22] P. Kuznetsova, V. Ordonez, A. C. Berg, T. L. Berg, and Y. Choi, “Collective generation of natural image descriptions,” in _Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1_. Association for Computational Linguistics, 2012, pp. 359–368.
  * [23] J. Devlin, S. Gupta, R. Girshick, M. Mitchell, and C. L. Zitnick, “Exploring nearest neighbor approaches for image captioning,” _arXiv preprint arXiv:1505.04467_ , 2015.
  * [24] P. Kuznetsova, V. Ordonez, T. Berg, and Y. Choi, “Treetalk: Composition and compression of trees for image descriptions,” _Transactions of the Association of Computational Linguistics_ , vol. 2, no. 1, pp. 351–362, 2014.
  * [25] M. Hodosh, P. Young, and J. Hockenmaier, “Framing image description as a ranking task: Data, models and evaluation metrics,” _Journal of Artificial Intelligence Research_ , vol. 47, pp. 853–899, 2013.
  * [26] A. Frome, G. S. Corrado, J. Shlens, S. Bengio, J. Dean, T. Mikolov _et al._ , “Devise: A deep visual-semantic embedding model,” in _Advances in neural information processing systems_ , 2013, pp. 2121–2129.
  * [27] R. Socher, A. Karpathy, Q. V. Le, C. D. Manning, and A. Y. Ng, “Grounded compositional semantics for finding and describing images with sentences,” _Transactions of the Association of Computational Linguistics_ , vol. 2, no. 1, pp. 207–218, 2014.
  * [28] A. Karpathy, A. Joulin, and L. F. Fei-Fei, “Deep fragment embeddings for bidirectional image sentence mapping,” in _Advances in neural information processing systems_ , 2014, pp. 1889–1897.
  * [29] X. Jia, E. Gavves, B. Fernando, and T. Tuytelaars, “Guiding the long-short term memory model for image caption generation,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2015, pp. 2407–2415.
  * [30] L. Ma, Z. Lu, L. Shang, and H. Li, “Multimodal convolutional neural networks for matching image and sentence,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 2623–2631.
  * [31] K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, R. Zemel, and Y. Bengio, “Show, attend and tell: Neural image caption generation with visual attention,” in _International Conference on Machine Learning_ , 2015, pp. 2048–2057.
  * [32] Q. Wu, C. Shen, L. Liu, A. Dick, and A. van den Hengel, “What value do explicit high level concepts have in vision to language problems?” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 203–212.
  * [33] J. Mao, W. Xu, Y. Yang, J. Wang, and A. L. Yuille, “Explain images with multimodal recurrent neural networks,” _CoRR_ , vol. abs/1410.1090, 2014\. [Online]. Available: http://arxiv.org/abs/1410.1090
  * [34] T. Yao, Y. Pan, Y. Li, Z. Qiu, and T. Mei, “Boosting image captioning with attributes,” _OpenReview_ , vol. 2, no. 5, p. 8, 2016.
  * [35] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, “Show and tell: Lessons learned from the 2015 mscoco image captioning challenge,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 39, no. 4, pp. 652–663, 2017.
  * [36] J. Donahue, L. Anne Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell, “Long-term recurrent convolutional networks for visual recognition and description,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 2625–2634.
  * [37] Q. Wu, C. Shen, P. Wang, A. Dick, and A. van den Hengel, “Image captioning and visual question answering based on attributes and external knowledge,” _IEEE transactions on pattern analysis and machine intelligence_ , 2017.
  * [38] J. Redmon and A. Farhadi, “Yolo9000: Better, faster, stronger,” in _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, 2017, pp. 6517–6525.
  * [39] M. Faruqui, J. Dodge, S. K. Jauhar, C. Dyer, E. Hovy, and N. A. Smith, “Retrofitting word vectors to semantic lexicons,” in _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 2015, pp. 1606–1615.
  * [40] L. Zhou, C. Xu, P. A. Koch, and J. J. Corso, “Image caption generation with text-conditional semantic attention,” _CoRR_ , vol. abs/1606.04621, 2016\. [Online]. Available: http://arxiv.org/abs/1606.04621
  * [41] T. Yao, Y. Pan, Y. Li, Z. Qiu, and T. Mei, “Boosting image captioning with attributes,” in _IEEE International Conference on Computer Vision, ICCV_ , 2017, pp. 22–29.
  * [42] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 2818–2826.
  * [43] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes (voc) challenge,” _International journal of computer vision_ , vol. 88, no. 2, pp. 303–338, 2010.
  * [44] X. Chen, H. Fang, T. Lin, R. Vedantam, S. Gupta, P. Dollár, and C. L. Zitnick, “Microsoft COCO captions: Data collection and evaluation server,” _CoRR_ , vol. abs/1504.00325, 2015. [Online]. Available: http://arxiv.org/abs/1504.00325


# LAB: Large-Scale Alignment for ChatBots

MIT-IBM Watson AI Lab and IBM Research
Shivchander Sudalairaj∗
Abhishek Bhandwaldar∗
Aldo Pareja∗
Kai Xu
David D. Cox
Akash Srivastava∗

*Equal Contribution

###### Abstract

This work introduces LAB (Large-scale Alignment for chatBots), a novel
methodology designed to overcome the scalability challenges in the
instruction-tuning phase of large language model (LLM) training. Leveraging a
taxonomy-guided synthetic data generation process and a multi-phase tuning
framework, LAB significantly reduces reliance on expensive human annotations
and proprietary models like GPT-4. We demonstrate that LAB-trained models can
achieve competitive performance across several benchmarks compared to models
trained with traditional human-annotated or GPT-4 generated synthetic data.
Thus offering a scalable, cost-effective solution for enhancing LLM
capabilities and instruction-following behaviors without the drawbacks of
catastrophic forgetting, marking a step forward in the efficient training of
LLMs for a wide range of applications.

##  1 Introduction

Large language models (LLMs) have achieved remarkable levels of success in
various natural language processing (NLP) applications, including question-
answering , entity extraction , and summarization . This has been made
possible, in large part, by the introduction of the transformer architecture ,
which can leverage large amounts of unlabeled, unstructured data, enabling the
scaling of LLMs to billions, or even trillions of parameters. LLMs are
typically trained in phases: a self-supervised pre-training phase, followed by
supervised alignment tuning phases.

The majority of the cost of training an LLM comes from the pre-training phase.
During this phase, a model is trained in an auto-regressive manner to predict
the next token in the target language using trillions of tokens worth of
unlabeled data, requiring thousands of GPUs training for months at a time.
Alignment tuning, typically happens in two stages: instruction tuning,
followed by preference tuning. Instruction tuning is more akin to the
traditional model training approach in machine learning, where the model is
trained directly on tasks of interest. In this stage, the model is given a
task description in the form of an natural language instuction (e.g. Summarize
the following news article in 2 lines: {News article}) and the model is
trained to maximize the likelihood of the provided ground truth summary.
Preference tuning, on the other hand, is done using techniques such as RLHF
(Stiennon et al., 2022; Ouyang et al., 2022) and DPO (Rafailov et al., 2023),
where the response from an instruction-tuned model is rated as preferred or
unpreferred using human feedback.

In comparison to pre-training, the instruction tuning and preference tuning
stages comprise a small fraction of the overall training procedure, both in
terms of the data used as well as the compute infrastructure required to train
models Touvron et al. (2023). For example, Meta’s LLaMA 2 models were trained
with just tens of thousands of high quality human-generated
instruction/response data pairs, followed by multiple rounds of RLHF with a
comparatively limited number of examples as compared to pretraining data
volumes Touvron et al. (2023). From a traditional machine learning training
perspective, this imbalance in the scale across the phases is
unconventional—typically one would expect a model to perform best when it has
been trained directly on the desired tasks, using as much data as possible.
The deviation from the tradtional LLM approach relies on the idea that
pretraining captures enough of the distribution of language and knowledge,
such that a small amount of supervised training can “unlock” or shape latent
abilities related to the ultimate desired instruction-following behavior of
the model. However, unlike the unstructured data that is abundantly available
in the public domain, high-quality, human-generated task-specific instruction
data is costly to procure, even via crowd-sourcing, and human-generated
instruction data is typically closely guarded by model builders, even for
ostensibly “open” model-building efforts. In this work, we address the
challenges associated with scaling of the alignment-tuning phase and propose a
new method called LAB: Large-scale Alignment for chatBots. The LAB method
consists of two components: (i) a taxonomy-guided synthetic data generation
method and quality assurance process that yields a highly diverse and high-
quality instruction dataset, without resorting to the use of proprietary LLMs
like GPT-4 or substantial human curation, and (ii) a novel multi-phase
training framework and unconventional tuning regime that allows for adding new
knowledge and instruction-following abilities into pre-trained LLMs without
suffering from catastrophic forgetting. Our findings show that LAB-trained
models can perform competitively with proprietary and open-source models that
use human annotations and/or synthetic data generated using GPT-4 on a number
of benchmarks.

##  2 Related Work

Existing methods for instruction tuning typically either rely on humans for
generating high-quality datasets, or use synthetic data generation using a
large teacher model. OpenAI (Ouyang et al., 2022) arguably set the standard
for model alignment from human data, employing human annotators to gather data
for supervised fine tuning (SFT) and reinforcement learning with human
feedback (RLHF) training. Collecting human-generated data for these steps is
complex undertaking; the selection of annotators requires a rigorous multi-
stage screening process aimed at achieving high inter-annotator agreement, and
collecting even modest amounts data (by LLM standards) requires the
coordination of large groups of annotators. The creators of the LLaMA 2 model
series (Touvron et al., 2023) followed a similar recipe, collecting tens of
thousands of human-generated instruction samples, and approximately 1 million
human-annotated binary comparisons for reward modeling. Not only are such
approaches expensive and time consuming, but they can also potentially limit
agility in exploring the space of instructions and capabilities the model is
trained to perform. Alternatives to this approach, such as transforming
existing human datasets into instructions via templating (Wei et al., ) can be
more cost effective, but face limitations in the naturalness and length of the
responses used for training.

More recently, training with synthetic data generated from LLMs has emerged as
an alternative to purely human-data-based approaches. Wang et al. (2023)
introduced Self-Instruct, which leverages a small number of handwritten human
seed instructions as input to bootstrapping process to generate a large number
of samples using an LLM’s own generation abilities. Taori et al. (2023) built
upon Self-Instruct, using a larger teacher model to generate synthetic data to
train a smaller student model, and incorporating principles in the generation
prompt to promote diversity in the generated instruction data. Xu et al.
(2023) introduces Evol-Instruct, another variant of Self-Instruct, that
synthesizes iteratively more complex instruction to overcome shortcomings of
previous methods. Mukherjee et al. (2023), Mitra et al. (2023) present a
synthetic data generation approach to enhance task diversity and scalability,
alongside a progressive training framework aimed at improving the model’s
reasoning ability and response style to match teacher models. This is achieved
by generating rich reasoning signals in the generated answer and progressively
training on datasets of varying difficulty in incremental phases.

Similar to LAB, concurrent work, GLAN (Li et al., 2024), employs a semi-
automatic approach to synthetic data generation that uses a human-curated
taxonomy to generate instruction tuning data from a teacher model. However, as
explained in section 3.2.2, unlike LAB, GLAN cannot be used to generate
synthetic data from domains that are not captured in the teacher model’s
support. As such, while LAB uses the open-source Mixtral model as the teacher,
like many other synthetic data generation approaches, GLAN has to rely on a
large proprietary model (GPT-4). This poses complicated questions about the
usability of generated data (especially for commercial purposes) since the
terms of use of proprietary models typically forbid using the model to improve
other models.

##  3 Methodology

LAB consists of two components: (i) a taxonomy to enable data curation
(section 3.1) as well as, guide the synthetic data generator (section 3.2) and
(ii) a multi-phased instruction-tuning method with replay buffers to enable
large-scale alignment-tuning. (section 3.3). (i) serves the purpose of
ensuring high diversity and quality in the synthetically generated
instruction-tuning dataset while (ii) ensures training stability and prevents
catastrophic forgetting. Figure 1 provides an overview of the end-to-end
pipeline of applying the LAB method to align a pre-trained LLM.

taxonomy rootknowledge$\dots$textbookfinanceexample 1example 2example
3$\dots$ptfoundationalskills$\dots$mathematicsarithmeticadditionmath instruct
data$\dots$ptcompositionalskills$\dots$writingemailearnings reportexample
1example 2example 3$\dots$document Synthetic Data Generator 1 Synthetic Data
Generator 2 synthetic “finance” data0.1–2ksynthetic “email” data0.1–2k Phased
training pre-trained LLMsection 3.1section 3.2section 3.3 Figure 1: Overview
of the LAB alignment method. Starting from the taxonomy root, data are curated
in each top-level groups and examples in the leaf nodes are used by the
synthetic data generators to generate orders of magnitude data for the phased-
training step for instruct-tuning.

###  3.1 Taxonomy

To enable the data curator or the model designer to organize the instruction-
tuning training data, we define a taxonomy that hierarchically classifies the
data samples into smaller task groups. At a high level, the taxonomy has three
main branches: knowledge, foundational skills, and compositional skills. Each
of these branches is further split into more granular levels where the tasks
are defined in the leaf nodes and exemplified by providing manually written
instruction-response pairs. This allows for easily identifying missing tasks
in the target LLM and other tasks of interest and adding them to the training
data pool. New tasks are added to the taxonomy by creating a leaf node under
the appropriate branch and attaching 1–3 examples.

##### Knowledge

The knowledge branch in the taxonomy is first divided based on document types
like textbooks, technical manuals, etc., which are further divided into
various domains like finance, statistics, etc.; see the sub-tree for knowledge
in Figure 1 as an example. Each domain has a collection of documents and a
sample set of domain-specific questions and answers. This organization allows
for better control over the licensing of text documents. As described in the
next section, only the documents with permissible licenses are selected for
synthetic data generation, excluding knowledge sources that lack proper
licensing, reinforcing the integrity of our knowledge-generation processes.

##### Foundational skills

We identify mathematics, coding, linguistic ability and reasoning as
foundational skills that the model requires to prime itself for better
knowledge acquisition and build further complex and compositional skills. To
teach the model foundational skills, we employ publicly available datasets
(Longpre et al., 2023; Xiang Yue, 2023; Yin et al., 2018; Trivedi et al.,
2022); see the sub-tree for foundational skills in Figure 1 for an example.

##### Compositional skills

Compositional skills refer to the tasks that require a combination of
knowledge and foundational skills, synergistically, to answer complex queries
from users. For instance, the model’s ability to write a company-wide email
sharing insights about the company’s performance last quarter and guidance for
the upcoming year would require the model to understand the financial aspects
of revenue, profit and loss, the skills of doing basic arithmetic and also
have the skills to compose a formal email.

###  3.2 Taxonomy-driven Synthetic Data Generator

The small number of manually curated data samples, embedded in the leaf nodes
of the taxonomy, can be directly used for instruction tuning of the chatbot,
however, the model may still perform poorly. Prior work (Li et al., 2023) has
shown that typically, a large amount of high-quality instruction data is
required for improving instruction following performance of LLMs. It is
possible to leverage existing SDGs like Wang et al. (2023); Taori et al.
(2023) to use the embedded examples and generate a lot more instruction data
synthetically using teacher LLMs. But, such distillation-based SDGs tend to
over-sample from the dominant modes of the teacher model and thus lack in
diversity and quality of the generated data Gudibande et al. (2023). We argue
that this limitation is attributed to the random selection of examples from
the pool of seed samples: with random selection, the examples used to prompt
the teacher model at each time are an “average” of the seed pool i.e. they do
not _focus_ on any specific task. This lack of focus tends to encourage the
teacher model to generate more synthetic data from its dominant modes and
ignore the long tail of interesting tasks.

To address this issue, we replace the random sampling in existing SDGs with a
taxonomy-driven approach to guide the sampling of synthetic data, enabling
targeted coverage of the support of the teacher model distribution around the
individual leaf nodes of the taxonomy. Figure 2 illustrate the high-level idea
behind this change.

(a) Input distributions

(b) Output distributions

Figure 2: Intuition of how taxonomy-driven sampling produces diverse set of
synthetic data and hence improve the data used to train student model across
the task domain. Figure 2(a) shows how taxonomy-driven sampling leads to an
input distribution with wide support and distinct modes while self-instruct
gives an smooth input distribution. Figure 2(b) shows the consequence using
inputs in generating synthetic data: teacher model will focus its own dominant
modes if the input is smooth but focus on each task better if the inputs are
also concentrated on each task.

Figure 2(a) shows the issue of randomly sampling in the _input_ space of the
teacher model (i.e. prompts). Given a set of seed examples (red), randomly
sampling with more than one example gives an approximation to the average of
the seed pool, leading to a smoothed distribution, e.g. self-instruct
distribution (blue). With the taxonomy-driven sampling, since only the
examples within each of the leaf nodes are used when sampling for the
corresponding tasks, each of the tasks are guaranteed to be well represented
in the prompts (purple). Second, when it comes to the _output_ space, for a
given teacher model (red), prompting it with random examples (i.e. smoothened
input distribution) tends to make it sampling from its own dominant mode
(blue) while prompting it with focused examples in each leaf node (i.e. input
distribution with distinct modes), the teacher model is guaranteed to generate
synthetic data for each of the tasks (purple).

With the above insight, we now introduce two new synthetic data generation
(SDG) methods in LAB that leverage the taxonomy to guide the data generation
process. The first one is targeted for skills generation and uses the handful
of task examples in the leaf nodes to generate a lot more using the open-
source Mixtral-7x8B model. The second one is targeted at knowledge generation.
While it still uses the Mixtral-7x8B model, unlike prior works, it does not
rely on the knowledge stored in the teacher model.

You are asked to come up with a set of {num_samples} diverse questions on
{task}. Please follow these guiding principles when generating responses: *
Use proper grammar and punctuation. * Always generate safe and respectful
content. Do not generate content that is harmful, abusive, or offensive. *
Always generate content that is factually accurate and relevant to the prompt.
* The questions should be clear and human-like. * The questions should be
diverse and cover a wide range of topics. * The questions should not be
template-based or generic, it should be very diverse. * Simply return the
questions, do not return any answers or explanations. * Strictly adhere to the
prompt and generate responses in the same style and format as the example. To
better assist you with this task, here is an example: ### Question: 1\.
{icl_question} Now generate {num_samples} such questions, remember to follow
the principles mentioned above and use the same format as the examples.
Remember to use the same style and format as the example above. Return your
responses in the format of [### Question [question number]: [question]] Figure
3: Instruction Generator prompt template

####  3.2.1 Skill Generation

Skills-SDG uses four prompt templates, one for each of the four, below-
mentioned, stages of data generation. Each template has its own set of
principles and instructions that control the role of the teacher model
(generator vs evaluator) and guide the generation/evaluation process.

  1. 1.

Instruction generation: In the first stage, the teacher model acts as a
question generator, using a specialized prompt (see Figure 3 for an example)
to leverage its knowledge and create diverse questions. By iterating through
each leaf node of a taxonomy, the teacher generates queries that adhere to
specific principles and thoroughly explore the targeted domain, enhancing the
comprehensiveness of the generated content.

  2. 2.

Evaluating synthetic instruction: In this stage, the teacher model assumes the
role of an instruction evaluator, the teacher model uses targeted prompts to
filter out questions that don’t meet predefined principles, including
relevance to the domain, potential harm, or questions beyond a language
model’s answering capabilities. This ensures that only high-quality,
contextually appropriate questions move forward in the process.

  3. 3.

Generating responses: The teacher model, functioning as a response generator
in this stage, adopts dual personas for precision and creativity, guided by
distinct prompts. This tailored approach helps to generate both, creative
responses for domains like writing and role-play, and precise answers for STEM
and data extraction, aligning the response style to human expectations through
principles and seed examples in the leaf nodes.

  4. 4.

Evaluating the synthetic instruction-response pair: The final stage involves a
rigorous process to filter and select high-quality instruction and response
pairs. Using a 3-point rating system (see Figure 4 for an example), the
teacher model evaluates each sample, filtering out those that are incorrect,
irrelevant, or deviate from the provided principles, ensuring the training
dataset’s quality and relevance are enhanced for the student model.

Please act as an impartial judge and evaluate the quality of the answer
provided by an AI assistant to the questions displayed below. Evaluate whether
or not the answer is a good example of how AI Assistant should respond to the
user’s instruction. Please assign a score using the following 3-point scale:
1: It means the answer is incorrect, irrelevant, unsafe or provides incomplete
and garbage information. For instance, the answer may be factually wrong, off-
topic, or filled with irrelevant content that doesn’t address the user’s
question or it could be incomplete and hanging. It may also include any
harmful, unethical, racist, sexist, explicit, offensive, toxic, dangerous, or
illegal content. 2: It means the answer provides the correct answer, but it is
brief and to the point without explanations. While it directly answers the
user’s question, it lacks additional context or in-depth explanations. 3: It
means the answer is a perfect answer from an AI Assistant. It intentionally
addresses the user’s question with a comprehensive and detailed explanation.
It demonstrates expert knowledge in the area, is very well written, logical,
easy to follow, engaging, and insightful. And the answer is safe and does not
include any harmful content. Begin your evaluation by providing a short
explanation. Be as objective as possible. After providing your explanation,
you must rate the answer on a scale of 1 to 3 as mentioned above. Please use
the following examples as a reference for your evaluation. Figure 4:
Instruction-response Evaluation template

####  3.2.2 Knowledge-generation

Synthetic data generators are inherently limited by the knowledge and
capabilities of the teacher model. This is one of the main reasons why most
successful SDG methods (Xu et al., 2023; Mukherjee et al., 2023; Mitra et al.,
2023) depend on GPT-4 model, which presumably has the highest coverage of
knowledge and skills. However, there are many domains that no open/proprietary
model is trained on and hence cannot work as a teacher model using existing
SDG methods. To address this limitation, in LAB we devised a new SDG pipeline
for generating instruction data on domains that the teacher model has not been
trained on. We call it knowledge-SDG.

Similar to the process of skills generation, knowledge-SDG uses the curator-
provided examples embedded in the leaf nodes of the knowledge branch of the
taxonomy. But additionally, the teacher model is provided a knowledge source
in the form of documents, manuals, and books on the target subject to ground
the generated instruction data into a reliable source thus avoiding dependence
on the internal knowledge base of a teacher model, which may struggle with
specialized domains and could lead to inaccuracies or hallucinations
especially on highly specialized, technical domains.

To ensure that the generated answers remain faithful to the content of the
source material, similar to the skills-SDG, teacher model is repurposed as an
evaluator that validates the generated responses are grounded and faithful to
the source documents.

###  3.3 Multi-Phase Training

LAB training happens in two phases, knowledge tuning, followed by skills
tuning.

In the knowledge-tuning phase, the model is trained on samples from the
knowledge and foundational skills branches of the taxonomy. This phase in-
turn, is carried out in two steps. We split the data under the knowledge and
foundational skills branches into two buckets based on the response length.
Then we first train the model on the samples with short responses before
moving on to training on samples with long responses. Similar to prior work
(Mitra et al., 2023), our empirical results also suggest that this two-step
approach to knowledge-tuning improves model performance.

Post-knowledge tuning, we start the skills-tuning phase where the best model
checkpoint from the knowledge-tuning phase is trained on the compositional
skills branch of the taxonomy. To address the challenge of catastrophic
forgetting when training in two distinct phases, a replay buffer of the data
from the knowledge-tuning phase in employed. Our empirical findings indicate
that starting with knowledge and foundational skills training, before
progressing to compositional skills leads to significantly better benchmark
performance.

For selecting the best model checkpoint during intermediate phases, we rely on
the MMLU benchmark (Hendrycks et al., 2020) during the knowledge-tuning phase
and the MT-bench (Zheng et al., 2024) during the skills-tuning phase. Please
refer to table 1 for an overview of our training phases.

##### Training Details

In our training process, we consciously avoid overtraining. Despite the
possibility of achieving higher scores on intermediate benchmarks, we have
found that selecting checkpoints from earlier stages of training results in
more reliable and generalizable model performance. We employ small learning
rates with an extended warm-up period, specifically $2\times 10^{-5}$ for
Llama-based models and $1\times 10^{-6}$ for Mistral-based models, each
beginning with a linear warm-up. This strategy is hypothesized to aid the
model in transitioning from broad dataset-wide learning to more focused, task-
specific adjustments. Additionally, we utilize a large effective batch size of
3840, achieved through gradient accumulation, to enhance stability across the
diverse range of tasks being learned concurrently. Our findings suggest that
using cosine decay on learning rates during intermediate phases can
destabilize subsequent training stages, likely due to the learning rate’s
reduction to near zero, narrowing the loss landscape and complicating the
integration of new phase gradients. Refer to table 2 for an overview of our
training hyper-parameters.

Phase | Step | Training data | Replay buffer |
---|---|---|---|---
Knowledge Tuning | 1 | Knowledge (short) | N/A |
2 |  Knowledge (long) Foundational skills | KT/1 data |
Skill Tuning | N/A | Compositional skills | KT/1 & KT/2 data |
Table 1: Data and reply buffers used in phase-training.

##  4 Results

In this study, we implemented the LAB method on two distinct open models,
Llama-2-13b (Touvron et al., 2023)and Mistral-7B (Jiang et al., 2023),
utilizing Mixtral-8x7B-Instruct-v0.1 (Jiang et al., 2024) as the teacher
model. This approach yielded two LAB-aligned models: Labradorite-13b and
Merlinite-7B.

During the synthetic data generation phase, we employed a taxonomy consisting
of numerous leaf nodes to produce a dataset comprising 1.2 million samples,
divided almost evenly between knowledge-based (617k) and skill-based (588k)
samples. The specific training hyper-parameters employed during this study are
summarized in table 2.

Model | Phase/step |  Learning rate |  Batch size |  Context length | #samples | #warm-up | #epochs
---|---|---|---|---|---|---|---
Labradorite-13b | KT/1 | 2e-5 | 3840 | 2048 | 630k | 385 | 5
KT/2 | 230k | 7
ST | 4096 | 550k | 7
Merlinite-7B | KT/1 | 1e-6 | 3840 | 2048 | 630k | 800 | 4
KT/2 | 230k | 4
ST | 4096 | 550k | 7
Table 2: Hyper-parameters used in training for Labradorite-13b and
Merlinite-7B.

We compare the performance of Labradorite-13b and Merlinite-7B against other
models that use the same base models for alignment, which include

##### Llama-2-13b

  * •

Llama-2-13b-chat (Touvron et al., 2023): RLHF with human annotators by the
same team that develops Llama-2-13b

  * •

Orca-2 (Mitra et al., 2023):

  * •

WizardLM-13B-V1.2 (Xu et al., 2023): model with the highest MT-Bench amongs
those use Llama-2-13b as the base model on LMSYS Chatbot Arena Leaderboard
(Zheng et al., 2023).

##### Mistral-7B

  * •

Mistral-7B-Instruct-v0.2 (Jiang et al., 2023): instruction-tuning using
supervised fine-tuning (SFT) on publicly available conversation datasets by
the same team that develops Mistral-7B

  * •

Zephyr-7b-beta (Tunstall et al., 2023): model with the highest MT-Bench amongs
those use Mistral-7B as the base model on LMSYS Chatbot Arena Leaderboard
(Zheng et al., 2023).

To compare the aligned LLMs, we consider the following evaluation metrics with
the settings consistent with those used by LMSYS Chatbot Arena Leaderboard
(Zheng et al., 2023)

  * •

MT-Bench (Zheng et al., 2023): 1-turn and 2-turn average

  * •

MMLU (Hendrycks et al., 2021): 5-shot

  * •

ARC (Clark et al., 2018): 25-shot

  * •

HellaSwag (Zellers et al., 2019): 10-shot

  * •

Winogrande (Sakaguchi et al., 2019): 5-shot

  * •

GSM8k (Cobbe et al., 2021): 5-shot strict

All results are reported in table 3.

Model | Alignment | Teacher |  MT-Bench |  MMLU |  ARC |  HellaSwag |  Winogrande |  GSM8K
---|---|---|---|---|---|---|---|---
Llama-2-13b-chat | SFT + RLHF |  Human annotators |  6.65† | 54.58 | 59.81 | 82.52 | 75.93 | 34.80
Orca-2 |  Progressive Training | GPT-4 |  6.15† | 60.37 | 59.73 | 79.86 | 78.22 | 48.22
WizardLM-13B |  Evol- Instruct | GPT-4 |  7.20† | 54.83 | 60.24 | 82.62 | 76.40 | 43.75
Labradorite-13b | LAB |  Mixtral-8x7B- Instruct |  7.23‡ | 58.89 | 61.69 | 83.15 | 79.56 | 40.11
Mistral-7B-Instruct | SFT |  Public Datasets |  6.84† | 60.37 | 63.65 | 84.76 | 76.80 | 41.85
Zephyr-7b-$\beta$ | SFT + DPO | GPT-4 |  7.34† | 61.07 | 63.74 | 84.19 | 78.06 | 34.04
Merlinite-7B | LAB |  Mixtral-8x7B- Instruct | 7.66‡ | 64.88 | 63.99 | 84.37 | 78.24 | 44.58

† taken from the LMSYS Chatbot Arena Leaderboard.
$\ddagger$ average of 3 runs.

Table 3: Evaluation of LLMs with different alignment methods over a
comprehensive set of benchmark metrics. Settings of each metric can be found
in the main text.

Notably, in terms of MT-Bench, Labradorite-13b performs better than the
current best model fine-tuned on Llama-2-13b and Merlinite-7B performs better
than the current best model fine-tuned on Mistral-7B, achieving state-of-the-
art performance in term of chatbot capability. Importantly, out training
method ensures that the model is not only good at multi-turn conversation but
also maintains its knowledge or reasoning capability, as shown by the overall
superior performance in the rest of the metrics. Besides, unlike those top
models that use GPT-4 as the teacher model, we achieve this performance using
the open-weights  Mixtral-8x7B-Instruct-v0.1, which is relatively weaker
teacher model at orders of magnitude less cost.

## References

  * Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord.  Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge, March 2018.
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman.  Training Verifiers to Solve Math Word Problems, November 2021.
  * Gudibande et al. (2023) Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song.  The false promise of imitating proprietary llms.  _arXiv preprint arXiv:2305.15717_ , 2023.
  * Hendrycks et al. (2020) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.  Measuring massive multitask language understanding.  In _International Conference on Learning Representations_ , 2020.
  * Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.  Measuring massive multitask language understanding, 2021.
  * Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed.  Mistral 7B, October 2023.
  * Jiang et al. (2024) Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed.  Mixtral of Experts, January 2024.
  * Li et al. (2024) Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, Yuxian Gu, Xin Cheng, Xun Wang, Si-Qing Chen, Li Dong, Wei Lu, Zhifang Sui, Benyou Wang, Wai Lam, and Furu Wei.  Synthetic data (almost) from scratch: Generalized instruction tuning for language models, 2024.
  * Li et al. (2023) Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Luke Zettlemoyer, Omer Levy, Jason Weston, and Mike Lewis.  Self-alignment with instruction backtranslation, 2023.
  * Longpre et al. (2023) Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts.  The flan collection: Designing data and methods for effective instruction tuning, 2023.
  * Mitra et al. (2023) Arindam Mitra, Luciano Del Corro, Shweti Mahajan, Andres Codas, Clarisse Simoes, Sahaj Agarwal, Xuxi Chen, Anastasia Razdaibiedina, Erik Jones, Kriti Aggarwal, Hamid Palangi, Guoqing Zheng, Corby Rosset, Hamed Khanpour, and Ahmed Awadallah.  Orca 2: Teaching Small Language Models How to Reason.  https://arxiv.org/abs/2311.11045v2, November 2023.
  * Mukherjee et al. (2023) Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah.  Orca: Progressive learning from complex explanation traces of gpt-4, 2023.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe.  Training language models to follow instructions with human feedback, 2022.
  * Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn.  Direct preference optimization: Your language model is secretly a reward model, 2023.
  * Sakaguchi et al. (2019) Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi.  WinoGrande: An Adversarial Winograd Schema Challenge at Scale, November 2019.
  * Stiennon et al. (2022) Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano.  Learning to summarize from human feedback, 2022.
  * Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca, 2023.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023.
  * Trivedi et al. (2022) Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal.  Musique: Multihop questions via single-hop question composition, 2022.
  * Tunstall et al. (2023) Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf.  Zephyr: Direct Distillation of LM Alignment, October 2023.
  * Wang et al. (2023) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi.  Self-Instruct: Aligning Language Models with Self-Generated Instructions, May 2023.
  * (22) Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le.  Finetuned language models are zero-shot learners.  In _International Conference on Learning Representations_.
  * Xiang Yue (2023) et al. Xiang Yue.  Mammoth: Building math generalist models through hybrid instruction tuning.  _arXiv preprint arXiv:2309.05653_ , 2023.
  * Xu et al. (2023) Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang.  WizardLM: Empowering Large Language Models to Follow Complex Instructions, June 2023.
  * Yin et al. (2018) Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan Vasilescu, and Graham Neubig.  Learning to mine aligned code and natural language pairs from stack overflow.  In _International Conference on Mining Software Repositories_ , MSR, pp. 476–486. ACM, 2018.  doi: https://doi.org/10.1145/3196398.3196408.
  * Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi.  HellaSwag: Can a Machine Really Finish Your Sentence?, May 2019.
  * Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica.  Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
  * Zheng et al. (2024) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al.  Judging llm-as-a-judge with mt-bench and chatbot arena.  _Advances in Neural Information Processing Systems_ , 36, 2024.