# Classifying FRB spectrograms using nonlinear dimensionality reduction
techniques

X. Yang1,2, S.-B. Zhang1, J.-S. Wang3, X.-F. Wu1,2
1 Purple Mountain Observatory, Chinese Academy of Sciences, Nanjing 210023,
China
2 School of Astronomy and Space Sciences, University of Science and Technology
of China, Hefei 230026, China
3 Max-Planck-Institut für Kernphysik, Saupfercheckweg 1, D-69117 Heidelberg,
Germany
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>(X-FW)

###### Abstract

Fast radio bursts (FRBs) are mysterious astronomical phenomena, and it is
still uncertain whether they consist of multiple types. In this study we use
two nonlinear dimensionality reduction algorithms - Uniform Manifold
Approximation and Projection (UMAP) and t-distributed stochastic neighbour
embedding (t-SNE) - to differentiate repeaters from apparently non-repeaters
in FRBs. Based on the first Canadian Hydrogen Intensity Mapping Experiment
(CHIME) FRB catalogue, these two methods are applied to standardized parameter
data and image data from a sample of 594 sub-bursts and 535 FRBs,
respectively. Both methods are able to differentiate repeaters from apparently
non-repeaters. The UMAP algorithm using image data produces more accurate
results and is a more model-independent method. Our result shows that in
general repeater clusters tend to be narrowband, which implies a difference in
burst morphology between repeaters and apparently non-repeaters. We also
compared our UMAP predictions with the CHIME/FRB discovery of 6 new repeaters,
the performance was generally good except for one outlier. Finally, we
highlight the need for a larger and more complete sample of FRBs.

###### keywords:

fast radio bursts < Transients, methods: data analysis < Astronomical
instrumentation, methods, and techniques

††pubyear: 2023††pagerange: Classifying FRB spectrograms using nonlinear
dimensionality reduction techniques–Classifying FRB spectrograms using
nonlinear dimensionality reduction techniques

##  1 Introduction

Fast Radio Bursts (FRBs) are bright radio pulses with a short duration of
typical milliseconds. Since the first FRB was discovered in Parkes archival
observation (Lorimer et al., 2007), more than 600 have been detected by
telescopes worldwide 111The FRB online Catalogue is available from www.wis-
tns.org. In general, FRBs have dispersion measures (DMs) that exceed what can
be attributed to our Galaxy (Yao et al., 2017), and they are considered to be
extragalactic. However, one FRB (FRB 200428) has been shown to be originated
from a Galactic magnetar (Bochenek et al., 2020; Andersen et al., 2020).
Furthermore, approximately 20 FRB sources have been localized to their host
galaxies to date (e.g. Chatterjee et al., 2017; Bannister et al., 2019; Ravi
et al., 2019; Prochaska et al., 2019; Macquart et al., 2020; Marcote et al.,
2020).

While the majority of FRBs are seen as one-off events, nearly 30 repeaters
have been identified (Spitler et al., 2016; Amiri et al., 2019; Andersen et
al., 2019; Kumar et al., 2019; Fonseca et al., 2020; Kirsten et al., 2022; Niu
et al., 2022; Xu et al., 2022), leading to the division of FRBs into two
classes based on their apparent repeatability. As the population of FRBs has
rapidly increased and their properties have been studied in more detail, a
better understanding of their classification has emerged. For example, Ravi
(2019) concluded that most FRB events must originate from repeaters based on
the evidence of their volumetric occurrence rate. Additionally, Lu et al.
(2020) found that the Canadian Hydrogen Intensity Mapping Experiment (CHIME)
repeater sample is consistent with the energy dependence of the apparently
non-repeating Australian Square Kilometre Array Pathfinder (ASKAP) sample,
suggesting that they may come from the same population.

Alternatively, with the rapid growth of the FRB samples, possible sub-
populations are gradually arising among the repeating or apparently non-
repeating FRBs. Most of the repeaters are highly linearly polarized (Michilli
et al., 2018; Nimmo et al., 2021), but FRB180301 shows a diversity of
polarimetric properties (Luo et al., 2020). Only two out of the nearly 30
repeaters show a periodicity (Rajwade et al., 2020; Chime/Frb Collaboration et
al., 2020). The repetition rates of some repeaters are reported much lower
than those notable repeating FRBs like FRB 20121102A and FRB 20201124A (Good
et al., 2022; Li et al., 2021; Xu et al., 2021). Guo & Wei (2022) apply
Kolmogorov-Smirnov (KS) test in CHIME/FRB catalogue, and introduce four new
sub-types of repeaters and apparently non-repeaters with respect to the cosmic
star formation history (SFH). However, the final confirmation of any sub-
populations for the repeating or apparently non-repeating FRBs still needs
more completed samples and higher sensitivity observations.

The CHIME/FRB Collaboration published its first FRB catalogue in 2021 (Amiri
et al., 2021), which is the first large-sample homogeneous FRB catalogue. The
catalogue includes 48 kinds of observation parameters and time-frequency
images of 536 bursts, including 474 apparently non-repeating bursts and 62
bursts from 18 repeating sources. To classify repeaters and apparently non-
repeaters in the catalogue, Chen et al. (2022) utilized an unsupervised
machine learning algorithm based on the Uniform Manifold Approximation and
Projection (UMAP). Their classification provides a repeating FRB completeness
of 95% and identifies 188 FRB repeater source candidates. This research
enlightens us about the importance of dimensionality reduction techniques for
accurately classifying FRBs.

Dimensionality reduction techniques are useful for visualizing high-
dimensional data. Principal component analysis (PCA) is a well-established
linear technique for dimensionality reduction that involves combining the
original variables to obtain principal components. However, nonlinear
techniques have been increasingly popular in recent years and are better
suited for addressing overcrowding problems in data feature extraction. The
t-Distributed Stochastic Neighbor Embedding (t-SNE) was the most commonly used
nonlinear technique in single-cell analysis before 2018 (Van der Maaten &
Hinton, 2008), but UMAP has since become more popular among scientists due to
its speed and ability to retain both local and global structures of data
(McInnes et al., 2018). In the field of biology, UMAP is known for providing
the fastest run times, highest reproducibility, and most meaningful
organization for cell clusters (Becht et al., 2019). In addition, proximity in
low dimensional UMAP space identifies groups of genes and finds novel protein
interactions (Dorrity et al., 2020). UMAP has also been proven efficient in
clustering astronomical data, including separating H$\alpha$-emission spectra,
randomly selecting spectra without H$\alpha$-emission (Sun et al., 2021), and
clustering auroral images (Lamb et al., 2019). In these researches, UMAP has
been applied to various input datasets, such as mass cytometry, single-Cell
RNA Sequencing, expression profile data, image and H$\alpha$-emission spectra.

In this paper, we utilize the datasets from the CHIME/FRB Catalogue to
investigate the morphology of FRBs and their classification using nonlinear
dimensionality reduction techniques. We apply these techniques to both the
time-frequency images and standardized high-dimensional observation parameters
of FRBs in order to identify potential sub-populations, and further explore
the impact of morphology on FRB classification. In Section 2, we provide a
description of the data and algorithms used in this study. The results of our
clustering analysis are presented in Section 3, while Section 4 includes a
discussion of our findings.

##  2 Implementation

###  2.1 Data selection

The data used in this study were obtained from the first CHIME/FRB catalogue
(Amiri et al., 2021), which includes 536 bursts or 600 sub-bursts detected in
the observations from July 25, 2018 to July 1, 2019. In instances where
multiple peaks appeared in the light curves, the FRBs were divided into sub-
bursts, resulting in a total of 600 sub-bursts. Of these, 506 sub-bursts were
identified in 474 apparently non-repeating FRBs, while 94 sub-bursts were
detected in 18 repeating FRBs.

To ensure comparability, the input data used for dimensionality reduction were
divided into two types: images with the same scale, and parameters converted
to the same scale. We followed the recommendation in the UMAP documentation to
convert each feature to the same scale.

(1) The first type of data used in this study were images containing the
dynamic spectrum of the dedispersed burst with time and frequency averaged. We
selected 535 FRB images, which included 599 FRB sub-bursts detected by the
CHIME/FRB Project 222The image of FRB20190626A is not provided by the
CHIME/FRB catalogue website.. These images were processed using _cfod_ python
package, and an example of the standard output is presented in the left panel
of Figure 1.

(2) The second type of data consisted of the parameters of the sub-bursts,
which included ten observed properties and three inferred properties. To
facilitate comparison with the work of Chen et al. (2022), we selected the
same parameters, namely: boxcar Width, width of sub-burst, flux, fluence,
scattering time, spectral index, spectral running, highest frequency, lowest
frequency, peak frequency, redshift, radio energy, and rest-frame intrinsic
duration. Since flux measurements were not available for six sub-bursts, this
study used 13 parameters for the 594 sub-bursts.

Figure 1: A sample of the extraction process of image data. The left panel
shows the _cfod_ standard output figure, the middle panel shows the extracted
burst area and the right panel shows the final input data for Resnet. Figure
2: The $256\times 256$ FRB image is utilized as the input image, and
subsequently passed into the pre-trained ResNet50V2 model. The final two
layers of the pretrained network have been substituted with an adaptive max
pooling layer, which is followed by two fully connected layer with 512 and 128
neurons, respectively. These two layers are activated by ReLU. The final
output is a fully connected layer with 2 neurons.

###  2.2 Data preparation

Each of the 535 image data generated by _cfod_ contains $911\times
870=792,570$ pixels. To avoid under-fitting, we extracted the burst area from
the image, as shown in the middle panel of Figure 1. In this process, we
extract multiple sub-bursts with full width at half maximum (FWHM) that
overlap each other as a single image. Seven of the 535 bursts have two
distinct sub-bursts, resulting in a total of 542 images. To augment the data,
we varied the width of each extracted image randomly along the time axis. As
shown in Figure 1, the cropped width of the middle image varied each time. For
each FRB, we generated five images with random widths between two to four
times the FWHM of the FRB, resulting in 2,710 images. We then employed a deep
residual network (ResNet) model to extract features from each image and obtain
a 512-dimensional feature vector that represents a compressed version of the
original image. The 2,710 images were resized to a dimension of 256x256 pixels
and fed into the ResNet model. The process we followed is similar to the
methodology proposed by Marianer et al. (2021). Given the limited number of
samples available for training, we opted for ResNet50V2, which contains 2.4
times fewer parameters than ResNet152V2. To fine-tune the pre-trained weight
of ResNet50V2, we used our labeled and augmented dataset comprising 1,200
images, including 600 labeled repeating FRBs and 600 labeled non-repeating
FRBs. As illustrated in Figure 2, we eliminated the final two layers and added
an adaptive max pooling layer, along with two fully connected layers
containing 512 and 128 neurons, respectively, activated by rectified linear
units (ReLU). Considering the presence of hidden repeaters in apparently non-
repeating FRBs, we did not expect the ResNet50V2 model to precisely predict
the labels. Following 100 training epochs, our ResNet50V2 implementation
achieved a best accuracy of 83.7% on the test set.

The selected thirteen parameters contain six kinds of units, three of which,
including spectral index, spectral running, and redshift are dimensionless. To
prevent the possible effect of parameter units in clustering results, instead
of using the original values of these parameters, we apply z-score
standardization (i.e., the number of standard deviations from the mean) to
them before training. This step is defaulted by the documentation of UMAP
333More detailed information can be found in umap-learn.readthedocs.io, and
this is the main difference between our input parameters and Chen et al.
(2022).

###  2.3 UMAP and t-SNE model configuration

We implemented python packages of the UMAP and t-SNE algorithms in this study.
UMAP is known to preserve global data structure well, while t-SNE is better at
presenting local data structure (Becht et al., 2019). Combining UMAP and t-SNE
allows us to gain a comprehensive understanding of the data from different
perspectives.

(1) To optimize the clustering results, we tuned three parameters for UMAP 3:
_n_neighbors_ , _min_dist_ , _n_components_. _n_neighbors_ represents the size
of the local neighbourhood and affects the balance between local and global
structure in the data. Larger values of _n_neighbors_ will make UMAP focus on
a broader structure of the data but may lose more detailed information.
_Min_dist_ controls the minimum distance between points in the low dimension
and helps to preserve the broad topological structure. _N_components_ allows
the user to choose the dimensionality of the reduced dimension, with two or
three dimensions being common in data visualization. We performed a grid
search for these parameters, _n_neighbors_ range from 2 to 100, _min_dist_
range from 0.01 to 0.99. Finally _n_neighbors=15_ , _min_dist=0.05_ and
_n_components=2_ are determined for the Resnet image feature dataset.
_n_neighbors=8_ , _min_dist=0.1_ and _n_components=2_ were determined for the
standardized parameter data.

(2) For t-SNE, we adjusted five parameters to optimize clustering 444More
detailed information can be found in scikit-learn.org: _perplexity_ ,
_early_exaggeration_ , _learning_rate_ , _n_iter_ , and _n_components_.
_Perplexity_ is similar to _n_neighbors_ in UMAP, affects the nearest
neighbours in the algorithm and is recommended to be set to larger values for
larger datasets. _Early_exaggeration_ changes the tightness of natural
clusters in the original embedded space and the space between natural
clusters. _Learning_rate_ controls the speed at which t-SNE updates its
parameters and can affect the amount of input information preserved in the
embedded data. _N_iter_ represents the number of iteration steps for the
optimization. As with UMAP, _n_components_ determines the dimensionality of
the reduced dimension. We performed a grid search for these parameters,
_perplexity_ range from 5 to 100, _early_exaggeration_ range from 10 to 80,
_learning_rate_ range from 10 to 1000. Finally we selected _perplexity=11_ ,
_early_exaggeration=46_ , _learning_rate=510_ , _n_iter=5000_ , and
_n_components=2_ for the Resnet image feature dataset, and _perplexity=15_ ,
_early_exaggeration=12_ , _learning_rate=200_ , _n_iter=5000_ , and
_n_components=2_ for the standardized parameter data. While parameters within
a reasonable range can yield acceptable results, the grid search helped us
choose the best parameters for our study.

##  3 Results and Discussion

###  3.1 UMAP and t-SNE training result

In Figure 3, the results of applying UMAP and t-SNE to the ResNet image
features of 542 FRBs are presented. The UMAP and t-SNE projections clearly
show that repeaters were tightly clustered together in a specific region,
while a noticeable gap was present between the mixture and pure non-repeaters.
These ResNet image features exhibited excellent performance in distinguishing
between repeaters and non-repeaters, indicating their potential as an
effective tool for identifying repeating FRBs.

For comparison with different input data, the FRB samples containing thirteen
standardized parameters are also reduced and clustered. Figure 4 shows the
UMAP and t-SNE embeddings of 594 FRB events. Although we used the same
thirteen parameters as Chen et al. (2022), our results are very different from
their separated distributions. Moreover, our results for the parameter data
are similar to those for the image data. Most of the repeaters are spread in a
quarter of the two-dimensional embedding plane, while the apparently non-
repeaters are randomly scattered on the whole plane. Both UMAP and t-SNE
successfully extract the difference between repeaters and apparently non-
repeaters in the outline.

Figure 3: The UMAP and t-SNE embeddings of 542 FRB Resnet image features. The
pink triangles represent repeaters, and the cyan dots represent non-repeaters.
Figure 4: The UMAP and t-SNE embeddings of 594 FRB standardized parameters.
The pink triangles represent repeaters, and the cyan dots represent non-
repeaters.

###  3.2 Classifying FRBs by the image and parameter data

Spectral clustering is one of the most popular modern clustering algorithms,
and clustering results obtained using this approach often outperform the
traditional approaches like k-means (Von Luxburg, 2007). It works by using the
eigenvectors of a similarity matrix to partition the data into clusters. In
this algorithm, the data points are treated as nodes in a graph, and
clustering is approached as a graph partitioning problem. Using the spectral
clustering algorithm, we clustered the UAMP two-dimensional embeddings
555Although t-SNE also seems to find the different distribution between two
kinds of FRBs, the structure of its embedding makes the spectral clustering
result unsatisfactory.. Based on the percentage of repeaters in the clusters,
we named them either repeater clusters or one-off clusters. The apparently
non-repeaters in the repeater cluster are labeled as repeater candidates.
Figure 5 shows the spectral clustering results of the image and parameter
data. To evaluate the performance of our model, we defined:

$\begin{split}{\rm Recall}=\frac{T_{\rm p}}{T_{\rm p}+F_{\rm n}},\end{split}$
(1)

where $T_{\rm p}$ is true positive, indicating the number of repeaters
correctly retrieved by the model in the repeater clusters, and $F_{\rm n}$ is
false negative, indicating the number of repeaters incorrectly retrieved by
the model in the apparently non-repeater clusters.

To optimize the clustering results, a grid search was performed with the
number of clusters ranging from 2 to 10. During this search, both the recall
and silhouette coefficient values were considered to evaluate the
effectiveness of the clustering. The Silhouette Coefficient is a measure of
how well-defined the clusters are, and it ranges between -1 and 1, with values
closer to 1 indicating more coherent clusters. A silhouette coefficient
greater than 0 indicates an acceptable result. The second row of Figure 5
displays the silhouette coefficient values of the clusters, with the red
dotted line indicating the average silhouette coefficient values, which were
greater than 0.4.

As shown in the middle panel of Figure 5, the UAMP embedding for Resnet image
feature data is clustered by the spectral clustering algorithm. The recall of
542 samples is 100.0%. Table 1 lists the detailed content of 7 clusters,
including 2 repeater clusters and 5 one-off clusters.

As shown in the right panel of Figure 5, the UAMP embedding for standardized
parameters data is also clustered by the spectral clustering algorithm. The
recall of 594 samples is 94.7%, with 5 repeater samples falling out of
repeater clusters. Table 2 lists the detailed content of 6 clusters, including
2 repeater clusters and 4 one-off clusters. From all 500 apparently non-
repeater sub-bursts, 145 repeater candidates were identified 666These sub-
burst candidates refer to 134 apparently non-repeating sources. Assuming they
are real repeaters, the FRB repeating rate is estimated to be 30.9%, lower
than the predicted rate of 41.9% by Chen et al. (2022).

Figure 5: The spectral clustering result of UMAP embedding. The label OC means One-off cluster and RC means Repeater cluster in table 1 and table 2. The second row are the corresponding silhouette coefficient values. The red dotted lines in the second row represent the average silhouette coefficient values of the clusters. Table 1: The information of the spectral clustering result of UMAP Resnet image feature dataset embedding. There are two repeater clusters, the apparently non-repeaters in repeater clusters are classified as repeater candidates. | Total number | Repeater number | Repeater candidate number | Apparently non-repeater number
---|---|---|---|---
One-off cluster 1 | 68 | 0 | 0 | 68
One-off cluster 2 | 79 | 0 | 0 | 79
One-off cluster 3 | 115 | 0 | 0 | 115
One-off cluster 4 | 47 | 0 | 0 | 47
One-off cluster 5 | 61 | 0 | 0 | 61
Repeater cluster 1 | 109 | 16 | 93 | 0
Repeater cluster 2 | 63 | 47 | 16 | 0
Table 2: The information of the spectral clustering result of UMAP standardized parameters dataset embedding. There are two repeater clusters, the apparently non-repeaters in repeater clusters are classified as repeater candidates. | Total number | Repeater number | Repeater candidate number | Apparently non-repeater number
---|---|---|---|---
One-off cluster 1 | 149 | 2 | 0 | 147
One-off cluster 2 | 15 | 0 | 0 | 15
One-off cluster 3 | 9 | 0 | 0 | 9
One-off cluster 4 | 187 | 3 | 0 | 184
Repeater cluster 1 | 141 | 40 | 101 | 0
Repeater cluster 2 | 93 | 49 | 44 | 0

According to the results presented, the Resnet image feature data performs
better among the two datasets in classifying FRBs. In addition to higher
recall, classifying FRBs using image data is more model-independent.

To better understand the clustering of the datasets, representative images of
each cluster were presented by stacking them together. For the Resnet image
feature data in Figure 6, it was found that narrowband FRBs of higher and
lower frequencies are mixed together in repeater clusters. Additionally, one-
off cluster 4, located on the left side of the gap between mixture and pure
non-repeaters, is composed mostly of narrowband FRBs. In Figure 7, for the
standardized parameters data, repeater cluster 1 represents lower frequency
narrowband FRBs, while repeater cluster 2 represents higher frequency
narrowband FRBs.

To investigate how different input types affect the UMAP clustering results,
we compared the FRB classification outcomes of two input types in our work
with one input type in Chen et al. (2022). Table 3 presents the detailed
comparison results. The spectral clustering of the Resnet image feature
dataset and thirteen standardized parameters classified 79.0% of apparently
non-repeaters with the same label. The spectral clustering of the Resnet image
feature dataset and the HDBSCAN clustering of thirteen parameters from Chen et
al. (2022) classified 71.7% of apparently non-repeaters with the same label.
The spectral clustering of thirteen standardized parameters and the HDBSCAN
clustering of thirteen parameter parameters classified 85.8% of apparently
non-repeaters with the same label.

Table 3: The clustering label similarity of apparently non-repeaters from four types of input. Dataset | Dataset | Label similarity
---|---|---
Resnet Image Feature | Standardized Parameters | 79.0%
Resnet Image Feature | Parameters | 71.7%
Standardized Parameters | Parameters | 85.8%
Figure 6: The representative images of each cluster for the Resnet image
feature. Each figure is obtained by stacking all the members in the specific
cluster. The detailed component information of each image is shown in Table 1.
Figure 7: The representative images of each cluster for the standardized
parameters. Each figure is obtained by stacking all the members in the
specific cluster. The detailed component information of each image is shown in
Table 2.

###  3.3 Feature importance

Figure 8: The left panel shows the result of permutation feature importance
for the Resnet image feature data, the right panel shows the result of
permutation feature importance for standardized parameters data.

To evaluate the contribution of each input image area or parameter,
permutation feature importance was used on the UMAP models, as described in
(Altmann et al., 2010). The values of each feature were randomly permuted, and
the resulting decrease in recall was used as an indicator of feature
relevance. The left panel of Figure 8 displays the permutation feature
importance for the Resnet image feature data, with a focus on the frequency
channel since the time length differs for each image. However, significant
fluctuation in the results was not found after permuting. The right panel of
Figure 8 presents the permutation feature importance of the thirteen
standardized observation parameters. The lowest frequency was found to be the
most important feature, influencing 18.9% of the final result, which is only
slightly more significant than the boxcar width, which impacts 17.5%. There is
no distinct boundary between the importance of different parameters, and no
single parameter dominates the clustering result.

###  3.4 Combination of spectrograms and physical parameters

536 sources were available with both spectrograms and physical parameters at
the same time. Additionally, we combined the array of images with standardized
parameters and used it as input for UMAP. Figure 9 shows the outcome of
combining these two types of image data with standardized parameters data.
However, there were no significant differences observed in the outputs
compared to those presented in Figure 3.

Figure 9: This figure shows the combined result of Resnet image feature and
standardized parameters.

##  4 Conclusion

Our findings suggest that UMAP and t-SNE, two nonlinear dimensionality
reduction models, have the potential to differentiate repeaters from
apparently non-repeaters in FRBs. We applied these models to the standardized
parameter data of 594 sub-bursts and 535 FRB images data (Amiri et al., 2021).
Both methods using two kinds of inputs were able to differentiate repeaters
from apparently non-repeaters. Our results do not reveal distinctly separated
clusters as in Chen et al. (2022) because we have followed the standardizing
process. In the Resnet image feature data, repeaters were tightly clustered
together in a specific region, while in the standardized parameter data, they
were spread in a quarter of the two-dimensional embedding plane. The
clustering recall using our standardized parameter data is 94.7%, which is
lower than the Resnet image feature data recall of 100.0%. These findings
suggest that classifying FRBs using image data is a more model-independent
method, with the potential to become a key method of future FRB
classification.

We compared our predictions with the CHIME/FRB discovery of 25 new repeaters,
and discovered that there were six repeaters in The CHIME/FRB Collaboration et
al. (2023) that were classified as apparently non-repeaters in Amiri et al.
(2021). The detailed information of our prediction for these repeaters is
listed in Table 4. We found that the standardized parameters dataset
successfully predicted five out of six FRBs, while the Resnet image feature
dataset predicted four out of six. It is worth noting that 20180910A appeared
to be an outlier for all methods because its spectrum resembled that of a
broadband FRB in the CHIME telescope, which only has a relatively narrowband
receiver (400-800 MHz).

Table 4:  The prediction of two kinds of dataset comparing to the six new repeaters (The CHIME/FRB Collaboration et al., 2023). The $\checkmark$ symbol means the FRB is predicted as repeater candidate, the $\times$ symbol means the FRB is predicted as apparently non-repeater.  FRB Source | Resnet Image Feature | Standardized Parameters
---|---|---
20190609C | ✓ | ✓
20190226B | $\times$ | ✓
20190430C | ✓ | ✓
20190110C | ✓ | ✓
20190113A | ✓ | ✓
20180910A | $\times$ | $\times$

Processing images is a convenient way to analyse data from different
instruments and at different frequency bands, such as Parkes, GBT, FAST, and
SKA. From representative images of each cluster, we found that repeater
clusters tend to be narrowband, which is consistent with our feature
importance analysis. These properties imply a difference in burst morphology
between repeaters and apparently non-repeaters. More FRBs detected by
instruments like the Parkes ultra-wide bandwidth (704 to 4032 MHz) receiver
(Hobbs et al., 2020) would be valuable for improving the accuracy of our
classification methods. Since the FRBs detected by CHIME are all within the
400-800 MHz range, our method can only classify FRBs within this frequency
range. However, when enough FRBs are detected at higher frequency bands (> 1
GHz), we can expand our classification method to cover these bands as well. If
the repeating and non-repeating types of FRBs are essential, our
classification methods should become more effective when applied to a larger
and more complete FRB sample.

## Acknowledgements

This work is partially supported by the National Key Research and Development
Program of China (2022SKA0130100), the National Natural Science Foundation of
China (grant No. 12041306), the international Partnership Program of Chinese
Academy of Sciences for Grand Challenges (114332KYSB20210018), the ACAMAR
Postdoctoral Fellow, China Postdoctoral Science Foundation (grant No.
2020M681758), and the Natural Science Foundation of Jiangsu Province (grant
Nos. BK20210998). JSW acknowledges the support from the Alexander von Humboldt
Foundation.

## DATA AVAILABILITY STATEMENTS

The FRB classifying results are available from
https://astroyx.github.io/frbdimensionreduction.github.io/. The data used in
this work is available from https://www.chime-frb.ca/catalog. The three
derived parameters are calculated following the method from Chen et al.
(2022).

## References

  * Altmann et al. (2010) Altmann A., Toloşi L., Sander O., Lengauer T., 2010, Bioinformatics, 26, 1340
  * Amiri et al. (2019) Amiri M., et al., 2019, Nature, 566, 235
  * Amiri et al. (2021) Amiri M., et al., 2021, ApJS, 257, 59
  * Andersen et al. (2019) Andersen B. C., et al., 2019, The Astrophysical Journal Letters, 885, L24
  * Andersen et al. (2020) Andersen B. C., et al., 2020, Nature, 587, 54
  * Bannister et al. (2019) Bannister K. W., et al., 2019, Science, 365, 565
  * Becht et al. (2019) Becht E., McInnes L., Healy J., Dutertre C.-A., Kwok I. W., Ng L. G., Ginhoux F., Newell E. W., 2019, Nature biotechnology, 37, 38
  * Bochenek et al. (2020) Bochenek C. D., Ravi V., Belov K. V., Hallinan G., Kocz J., Kulkarni S. R., McKenna D. L., 2020, Nature, 587, 59
  * Chatterjee et al. (2017) Chatterjee S., et al., 2017, Nature, 541, 58
  * Chen et al. (2022) Chen B. H., Hashimoto T., Goto T., Kim S. J., Santos D. J. D., On A. Y. L., Lu T.-Y., Hsiao T. Y. Y., 2022, MNRAS, 509, 1227
  * Chime/Frb Collaboration et al. (2020) Chime/Frb Collaboration et al., 2020, Nature, 582, 351
  * Dorrity et al. (2020) Dorrity M. W., Saunders L. M., Queitsch C., Fields S., Trapnell C., 2020, Nature communications, 11, 1
  * Fonseca et al. (2020) Fonseca E., et al., 2020, ApJ, 891, L6
  * Good et al. (2022) Good D. C., et al., 2022, arXiv e-prints, p. arXiv:2204.09090
  * Guo & Wei (2022) Guo H.-Y., Wei H., 2022, arXiv e-prints, p. arXiv:2203.12551
  * Hobbs et al. (2020) Hobbs G., et al., 2020, Publ. Astron. Soc. Australia, 37, e012
  * Kirsten et al. (2022) Kirsten F., et al., 2022, Nature, 602, 585
  * Kumar et al. (2019) Kumar P., et al., 2019, The Astrophysical Journal Letters, 887, L30
  * Lamb et al. (2019) Lamb K., et al., 2019, arXiv e-prints, p. arXiv:1910.03085
  * Li et al. (2021) Li D., et al., 2021, Nature, 598, 267
  * Lorimer et al. (2007) Lorimer D., Bailes M., McLaughlin M., Narkevic D., Crawford F., 2007, Science, 318, 777
  * Lu et al. (2020) Lu W., Piro A. L., Waxman E., 2020, MNRAS, 498, 1973
  * Luo et al. (2020) Luo R., et al., 2020, Nature, 586, 693
  * Macquart et al. (2020) Macquart J. P., et al., 2020, Nature, 581, 391
  * Marcote et al. (2020) Marcote B., et al., 2020, Nature, 577, 190
  * Marianer et al. (2021) Marianer T., Poznanski D., Prochaska J. X., 2021, MNRAS, 500, 5408
  * McInnes et al. (2018) McInnes L., Healy J., Melville J., 2018, arXiv e-prints, p. arXiv:1802.03426
  * Michilli et al. (2018) Michilli D., et al., 2018, Nature, 553, 182
  * Nimmo et al. (2021) Nimmo K., et al., 2021, Nature Astronomy, 5, 594
  * Niu et al. (2022) Niu C. H., et al., 2022, Nature, 606, 873
  * Prochaska et al. (2019) Prochaska J. X., et al., 2019, Science, 366, 231
  * Rajwade et al. (2020) Rajwade K. M., et al., 2020, MNRAS, 495, 3551
  * Ravi (2019) Ravi V., 2019, Nature Astronomy, 3, 928
  * Ravi et al. (2019) Ravi V., et al., 2019, Nature, 572, 352
  * Spitler et al. (2016) Spitler L., et al., 2016, Nature, 531, 202
  * Sun et al. (2021) Sun Y., et al., 2021, ApJS, 257, 65
  * The CHIME/FRB Collaboration et al. (2023) The CHIME/FRB Collaboration et al., 2023, arXiv e-prints, p. arXiv:2301.08762
  * Van der Maaten & Hinton (2008) Van der Maaten L., Hinton G., 2008, Journal of machine learning research, 9
  * Von Luxburg (2007) Von Luxburg U., 2007, Statistics and computing, 17, 395
  * Xu et al. (2021) Xu H., et al., 2021, arXiv e-prints, p. arXiv:2111.11764
  * Xu et al. (2022) Xu H., et al., 2022, Nature, 609, 685
  * Yao et al. (2017) Yao J. M., Manchester R. N., Wang N., 2017, The Astrophysical Journal, 835, 29


$\tau_{\mathcal{U}}(T)=\lim_{k\to\mathcal{U}}\langle Tf_{k},f_{k}\rangle$ is a
pure state which is singular if and only if $\mathcal{U}$ is non-principal
(cf. Theorem 4.21 and Example 6.1 in [15]). Such pure states are said to be
_diagonalizable_. On an abelian C*-algebra, pure states coincide with
characters, so the aforementioned $\tau_{\mathcal{U}}$ restricts to a pure
state on the atomic maximal abelian self-adjoint subalgebra (or _masa_)
generated by the rank-one projections corresponding to the $f_{k}$. The
following problem asks to what extent this is true of all pure states:

######  Problem (Kadison–Singer [21]).

Does every pure state on $\mathcal{B}(H)$ restrict to a pure state on some
(atomic or continuous) masa?

Anderson conjectured that not only is the answer to this question “yes”, but
that every pure state is of the form $\tau_{\mathcal{U}}$, for some choice of
orthonormal basis $(f_{k})$ and ultrafilter $\mathcal{U}$:

######  Conjecture (Anderson [3]).

Every pure state on $\mathcal{B}(H)$ is diagonalizable.

Akemann and Weaver [1] showed that the above problem of Kadison and Singer has
a negative answer, and thus Anderson’s conjecture is false, assuming
$\mathsf{CH}$. It remains an open question whether Anderson’s conjecture is
consistent with $\mathsf{ZFC}$.

By the recent positive solution [29] to the Kadison–Singer problem regarding
_extensions_ of pure states (which differs from the above), Anderson’s
conjecture is equivalent to saying that every pure state on $\mathcal{B}(H)$
restricts to a pure state on some atomic masa.

Following [8], we say that a subset
$\mathcal{F}\subseteq\mathcal{P}(\mathcal{C}(H))^{+}$ is _centered_
101010These were called _quantum filters_ by Farah and Weaver [15]. if for
every finite subset of $\mathcal{F}$ has a lower bound in
$\mathcal{P}(\mathcal{C}(H))^{+}$. $\mathcal{F}$ is _linked_ if every pair of
elements in $\mathcal{F}$ has a lower bound in
$\mathcal{P}(\mathcal{C}(H))^{+}$. _Maximal centered_ has the obvious meaning.
Similarly, we define _$\leq_{\mathrm{ess}}$ -centered_, _$\leq_{\mathrm{ess}}$
-linked_, and _maximal $\leq_{\mathrm{ess}}$-centered_ in
$\mathrm{bb}^{\infty}_{1}(E)$.

######  Theorem 9.8 (Farah–Weaver, Theorem 6.42 in [15]).

There is a bijective correspondence between singular pure states $\tau$ on
$\mathcal{B}(H)$ and maximal centered subsets of
$\mathcal{P}(\mathcal{C}(H))^{+}$ via
$\tau\mapsto\mathcal{F}_{\tau}=\\{p\in\mathcal{P}(\mathcal{C}(H))^{+}:\tau(p)=1\\}$.

If $\mathcal{F}=\mathcal{F}_{\tau}$ as above and $\tau$ fails to restrict to a
pure state on any atomic masa, we say that $\mathcal{F}$ _yields a
counterexample to Anderson’s conjecture_.

######  Theorem 9.9 (essentially Farah–Weaver, cf. Theorem 6.46 in [15]).

If $\mathcal{G}$ is $\mathbf{V}$-generic for
$\mathcal{P}(\mathcal{C}(H))^{+}$, then $\mathcal{G}$ is a maximal centered
set which yields a counterexample to Anderson’s conjecture.

In fact, this result uses much less than full genericity, or even genericity
over $\mathbf{L}(\mathbb{R})$. By considering the complexity of the dense sets
involved in the proof, we obtain Theorem 1.7:

###### Proof of Theorem 1.7..

Let $\mathcal{H}\subseteq\mathrm{bb}^{\infty}_{1}(E)$ be spread
$(p^{*})$-family which is $\leq_{\mathrm{ess}}$-centered and
$\widehat{\mathcal{H}}$ the upwards closure of $\pi(\mathcal{H})$ in
$\mathcal{P}(\mathcal{C}(H))^{+}$. First, we claim that
$\widehat{\mathcal{H}}$ is a maximal centered set. Clearly,
$\widehat{\mathcal{H}}$ is centered. For maximality, let
$p\in\mathcal{P}(\mathcal{C}(H))^{+}$ be such that $p$ is compatible with
every finite subset of $\widehat{\mathcal{H}}$. Let $P\in\mathcal{P}(H)$ be
such that $\pi(P)=p$, and define

$\mathcal{D}_{P}=\\{X:P_{X}\leq_{\mathrm{ess}}P\text{ or
}P_{X}\bot_{\mathrm{ess}}P\\},$

which is a co-analytic and $\leq_{\mathrm{ess}}$-dense open subset of
$\mathrm{bb}^{\infty}_{1}(H)$. By Lemma 9.6, $\mathcal{D}_{P}$ is
$\preceq$-dense open, so by Theorem 8.14, we can find a
$Y\in\mathcal{H}\upharpoonright X$ with $Y\in\mathcal{D}_{P}$. It must then be
the case that $P_{Y}\leq_{\mathrm{ess}}P$ and so $p\in\widehat{\mathcal{H}}$.

To see that $\widehat{\mathcal{H}}$ yields a counterexample to Anderson’s
conjecture, we refer to the proof of Theorem 6.46 in [15] and omit the details
except to note that it suffices to show that $\mathcal{H}$ meets the
$\leq_{\mathrm{ess}}$-dense open sets

$\mathcal{D}_{\vec{J}}=\\{X\in\mathrm{bb}^{\infty}_{1}(E):\forall
n(\|P^{(f_{k})}_{J_{n}\cup J_{n+1}}P_{X}\|<1/2)\\},$

where $\vec{J}=(J_{n})$ is a partition of $\mathbb{N}$ into finite intervals
$J_{n}$ and $P^{(f_{k})}_{J}$ denotes the orthogonal projection onto
$\overline{\operatorname{span}}\\{f_{k}:k\in J\\}$, for $(f_{k})$ an
orthonormal basis of $H$. These sets are easily seen to be Borel, and meeting
them with $\mathcal{H}$ uses the combination of Lemma 9.6 and Theorem 8.14 as
before. ∎

For spread $(p^{*})$-families, being $\leq_{\mathrm{ess}}$-linked implies
being a $\leq_{\mathrm{ess}}$-filter:

######  Lemma 9.10.

Let $\mathcal{H}\subseteq\mathrm{bb}^{\infty}_{1}(E)$ be a spread
$(p^{*})$-family which is, moreover, $\leq_{\mathrm{ess}}$-linked. Then,
$\mathcal{H}$ is a $\leq_{\mathrm{ess}}$-filter.

###### Proof.

Let $X,Y\in\mathcal{H}$, and consider the set

$\mathcal{D}=\\{Z:(Z\leq_{\mathrm{ess}}X\text{ and
}Z\leq_{\mathrm{ess}}Y)\text{ or }(Z\bot_{\mathrm{ess}}X\text{ or
}Z\bot_{\mathrm{ess}}Y)\\}.$

It is easy to check that $\mathcal{D}$ is co-analytic. Clearly $\mathcal{D}$
is $\leq_{\mathrm{ess}}$-dense open, thus $\preceq$-dense open by Lemma 9.6.
By Theorem 8.14 applied to the analytic set $\mathbb{A}=\mathcal{D}^{c}$,
there is a $Z\in\mathcal{H}$ with $[Z]_{1}\subseteq\mathcal{D}$. In
particular, $Z\in\mathcal{D}$. Since $\mathcal{D}$ is
$\leq_{\mathrm{ess}}$-linked, we must have that $Z\leq_{\mathrm{ess}}X$ and
$Z\leq_{\mathrm{ess}}Y$. ∎

By Lemma 9.10, the maximal centered sets in Theorem 1.7 are also filters in
$\mathcal{P}(\mathcal{C}(H))^{+}$. The following result of Bice, using
Shelah’s model without $p$-points (VI. §4 in [36]), presents an obstacle to
$\mathsf{ZFC}$ constructions.

######  Theorem 9.11 (Bice [8]).

It is consistent with $\mathsf{ZFC}$ that no maximal centered set in
$\mathcal{P}(\mathcal{C}(H))^{+}$ is a filter.

Consequently, we have:

######  Corollary 9.12.

It is consistent with $\mathsf{ZFC}$ that no spread $(p^{*})$-family in
$\mathrm{bb}^{\infty}_{1}(E)$ can be $\leq_{\mathrm{ess}}$-linked, and in
particular, that there are no spread $(p^{*})$-filters.

##  10\. Further questions

Despite our constructions, under additional hypotheses, of $(p^{+})$-filters,
there remains a lack of examples of interesting, purely analytical, $(p^{+})$
and $(p^{*})$-families, Example 8.27 notwithstanding.

######  Question.

Are there naturally occurring non-trivial ($\mathsf{ZFC}$) examples of
$(p^{+})$ or $(p^{*})$ families of block sequences?

While Theorem 1.6 does give a criterion for
$\mathbf{L}(\mathbb{R})$-genericity for filters of projections in the Calkin
algebra, it would be desirable to have a such criterion expressed in the
language of C*-algebras.

######  Question.

Can the (local) Ramsey theory of block sequences in a separable infinite-
dimensional Hilbert space be described in C*-algebraic terms? Under large
cardinals, is there a C*-algebraic characterization of
$\mathbf{L}(\mathbb{R})$-generic filters in the projections in the Calkin
algebra?

Lastly, as the sufficient conditions described in Theorem 1.7 for producing a
counterexample to Anderson’s conjecture cannot be satisfied in Shelah’s model
without $p$-points, the status of Anderson’s conjecture in that model appears
to be a natural test question.

######  Question.

Does Anderson’s conjecture hold in Shelah’s model without
$p$-points?111111Added in proof: This question has a negative answer. It
follows from the fact that Anderson’s conjecture fails whenever $\mathfrak{d}$
is less than or equal to the analogue of $\mathfrak{t}$ for
$\mathcal{P}(\mathcal{C}(H))$ [15], which holds in Shelah’s model.

## References

  * [1] C. Akemann and N. Weaver.  $\mathcal{B}(H)$ has a pure state that is not multiplicative on any masa.  Proc. Natl. Acad. Sci. USA, 105(14):5313–5314, 2008.
  * [2] F. Albiac and N. J. Kalton.  Topics in Banach space theory, volume 233 of Graduate Texts in Mathematics.  Springer, New York, 2006.
  * [3] J. Anderson.  A conjecture concerning the pure states of $\mathcal{B}(H)$ and a related theorem.  In Topics in modern operator theory (Timişoara/Herculane, 1980), volume 2 of Operator Theory: Adv. Appl., pages 27–43. Birkhäuser, Basel-Boston, Mass., 1981.
  * [4] S. A. Argyros and S. Todorčević.  Ramsey methods in analysis.  Advanced Courses in Mathematics. CRM Barcelona. Birkhäuser Verlag, Basel, 2005.
  * [5] J. Bagaria and J. López-Abad.  Weakly Ramsey sets in Banach spaces.  Adv. Math., 160(2):133–174, 2001.
  * [6] J. Bagaria and J. López-Abad.  Determinacy and weakly Ramsey sets in Banach spaces.  Trans. Amer. Math. Soc., 354(4):1327–1349 (electronic), 2002.
  * [7] T. Bartoszyński and H. Judah.  Set theory: On the structure of the real line.  A K Peters, Ltd., Wellesley, MA, 1995.
  * [8] T. M. Bice.  Filters in $\rm C^{*}$-algebras.  Canad. J. Math., 65(3):485–509, 2013.
  * [9] A. Blass.  Ultrafilters related to Hindman’s finite-unions theorem and its extensions.  In Logic and combinatorics (Arcata, Calif., 1985), volume 65 of Contemp. Math., pages 89–124. Amer. Math. Soc., Providence, RI, 1987.
  * [10] A. Blass.  Combinatorial cardinal characteristics of the continuum.  In Handbook of set theory. Vol. 1, pages 395–489. Springer, Dordrecht, 2010.
  * [11] A. Blass and N. Hindman.  On strongly summable ultrafilters and union ultrafilters.  Trans. Amer. Math. Soc., 304(1):83–97, 1987.
  * [12] C. A. Di Prisco, J. G. Mijares, and J. Nieto.  Local ramsey theory. An abstract approach.  To appear in MLQ Math. Log. Q.
  * [13] C. A. Di Prisco and S. Todorčević.  Souslin partitions of products of finite sets.  Adv. Math., 176(1):145–173, 2003.
  * [14] I. Farah.  Semiselective coideals.  Mathematika, 45(1):79–103, 1998.
  * [15] I. Farah and E. Wofsey.  Set theory and operator algebras.  In J. Cummings and E. Schimmerling, editors, Appalachian Set Theory 2006-2012, number 406 in Lecture Notes Series, pages 63–120. London Mathematical Society, 2012.
  * [16] Q. Feng, M. Magidor, and W. H. Woodin.  Universally Baire sets of reals.  In Set theory of the continuum (Berkeley, CA, 1989), volume 26 of Math. Sci. Res. Inst. Publ., pages 203–242. Springer, New York, 1992.
  * [17] V. Ferenczi and C. Rosendal.  Ergodic Banach spaces.  Adv. Math., 195(1):259–282, 2005.
  * [18] W. T. Gowers.  A new dichotomy for Banach spaces.  Geom. Funct. Anal., 6(6):1083–1093, 1996.
  * [19] W. T. Gowers.  An infinite Ramsey theorem and some Banach-space dichotomies.  Ann. of Math. (2), 156(3):797–833, 2002.
  * [20] N. Hindman.  Finite sums from sequences within cells of a partition of $\mathbb{N}$.  J. Combinatorial Theory Ser. A, 17:1–11, 1974.
  * [21] R. V. Kadison and I. M. Singer.  Extensions of pure states.  Amer. J. Math., 81:383–400, 1959.
  * [22] A. S. Kechris.  Classical descriptive set theory, volume 156 of Graduate Texts in Mathematics.  Springer-Verlag, New York, 1995.
  * [23] R. A. Komorowski and N. Tomczak-Jaegermann.  Banach spaces without local unconditional structure.  Israel J. Math., 89(1-3):205–226, 1995.
  * [24] K. Kunen.  Some points in $\beta N$.  Math. Proc. Cambridge Philos. Soc., 80(3):385–398, 1976.
  * [25] C. Laflamme.  Forcing with filters and complete combinatorics.  Ann. Pure Appl. Logic, 42(2), 125–163, 1989.
  * [26] C. Laflamme, L. Nguyen Van Thé, M. Pouzet, and N. Sauer.  Partitions and indivisibility properties of countable dimensional vector spaces.  J. Combin. Theory Ser. A, 118(1):67–77, 2011.
  * [27] P. B. Larson.  The stationary tower: Notes on a course by W. Hugh Woodin, volume 32 of University Lecture Series.  Amer. Math. Soc., Providence, RI, 2004.
  * [28] J. López-Abad.  Coding into Ramsey sets.  Math. Ann., 332(4):775–794, 2005.
  * [29] A. W. Marcus, D. A. Spielman, and N. Srivastava.  Interlacing families II: Mixed characteristic polynomials and the Kadison-Singer problem.  Ann. of Math. (2), 182(1):327–350, 2015.
  * [30] A. R. D. Mathias.  Happy families.  Ann. Math. Logic, 12(1):59–111, 1977.
  * [31] J. G. Mijares.  A notion of selective ultrafilter corresponding to topological Ramsey spaces.  MLQ Math. Log. Q., 53(3):255–267, 2007.
  * [32] K. R. Milliken.  Ramsey’s theorem with sums or unions.  J. Combinatorial Theory Ser. A, 18:276–290, 1975.
  * [33] J. T. Moore.  The method of forcing.  In preperation, 2016.
  * [34] C. Rosendal.  Infinite asymptotic games.  Ann. Inst. Fourier (Grenoble), 59(4):1359–1384, 2009.
  * [35] C. Rosendal.  An exact Ramsey principle for block sequences.  Collect. Math., 61(1):25–36, 2010.
  * [36] S. Shelah.  Proper and improper forcing.  Perspectives in Mathematical Logic. Springer-Verlag, Berlin, second edition, 1998.
  * [37] S. Shelah and W. H. Woodin.  Large cardinals imply that every reasonably definable set of reals is Lebesgue measurable.  Israel J. Math., 70(3):381–394, 1990.
  * [38] J. Silver.  Every analytic set is Ramsey.  J. Symbolic Logic, 35:60–64, 1970.
  * [39] I. B. Smythe.  Set theory in infinite-dimensional vector spaces.  PhD thesis, Cornell University, 2017.
  * [40] S. Todorčević.  Introduction to Ramsey spaces, volume 174 of Annals of Mathematics Studies.  Princeton University Press, Princeton, NJ, 2010.
  * [41] N. Weaver.  Set theory and $C^{*}$-algebras.  Bull. Symbolic Logic, 13(1):1–20, 2007.
  * [42] Y. Y. Zheng.  Selective ultrafilters on $\mathrm{FIN}$.  Proc. Amer. Math. Soc., 145(12):5071–5086, 2017.


#  Flacuna: Unleashing the Problem Solving Power of Vicuna using Flan Fine-
Tuning

Deepanway Ghosal‡, Yew Ken Chia‡, Navonil Majumder†, Soujanya Poria‡
‡ DeCLaRe Lab, Singapore University of Technology and Design, Singapore
{deepanway_ghosal<EMAIL_ADDRESS>
<EMAIL_ADDRESS>

###### Abstract

Recently, the release of InstructEval (Chia et al., 2023) has provided
valuable insights into the performance of large language models (LLMs) that
utilize encoder-decoder or decoder-only architecture. Interestingly, despite
being introduced four years ago, T5-based LLMs, such as Flan-T5, continue to
outperform the latest decoder-based LLMs, such as LLaMA and Vicuna, on tasks
that require general problem-solving skills. This performance discrepancy can
be attributed to three key factors: (1) Pre-training data, (2) Backbone
architecture, and (3) Instruction dataset. In this technical report, our main
focus is on investigating the impact of the third factor by leveraging Vicuna,
a large language model based on LLaMA, which has undergone fine-tuning on
ChatGPT conversations. To achieve this objective, we fine-tuned Vicuna using a
customized instruction dataset collection called Flan-mini. This collection
includes a subset of the large-scale instruction dataset known as Flan, as
well as various code-related datasets and conversational datasets derived from
ChatGPT/GPT-4. This dataset comprises a large number of tasks that demand
problem-solving skills. Our experimental findings strongly indicate that the
enhanced problem-solving abilities of our model, Flacuna, are obtained through
fine-tuning Vicuna on the Flan dataset, leading to significant improvements
across numerous benchmark datasets in InstructEval. Flacuna is publicly
available at https://huggingface.co/declare-lab/flacuna-13b-v1.0.

Code: https://github.com/declare-lab/flacuna

Model: https://huggingface.co/declare-lab/flacuna-13b-v1.0

Flan-mini: https://huggingface.co/declare-lab/flan-mini

##  1 Introduction

ChatGPT and its successor GPT-4 have surpassed their prior state-of-the-art
models on a vast majority of the benchmarking tasks and datasets. However, to
preserve privacy, natively running a 175B+ sized model like GPT-3 is beyond
the capabilities of most organizations, let alone individuals. This has
prompted many researchers to fine-tune manageable-sized LLMs — from 7B to 30B
on a diverse set of instruction examples generated by ChatGPT or GPT-4. This
has birthed LLMs, such as, Alpaca (Taori et al., 2023) and Vicuna (Chiang et
al., 2023) that are fine-tuned checkpoints of LLaMA (Touvron et al., 2023).
These models have attained close to ChatGPT-level performance on some specific
benchmarking tasks, but overall generalization still remains elusive. Recent
works like InstructEval (Chia et al., 2023) strongly hint that the fine-tuning
datasets dictate the task-specific performances. For instance, it has been
observed that Flan-T5 — a T5 checkpoint fine-tuned on Flan Collection
instruction dataset — outperforms Vicuna and Alpaca on tasks involving strong
reasoning and problem-solving skills. This spurred us to fine-tune Vicuna on
Flan-mini Collection dataset, anticipating improvement on reasoning-intensive
tasks in InstructEval (Chia et al., 2023).

To this end, we first sample a 1M-sized instruction dataset from the 15M-sized
Flan Collection dataset (Longpre et al., 2023) and combined it with several
other datasets comprising coding tasks and ChatGPT/GPT-4 distilled
conversations. The resulting smaller dataset, Flan-mini, is then cast into the
conversational format of Vicuna. To ensure a reasonable computational cost for
the fine-tuning process, we retrofit LoRA (Hu et al., 2021) adapter into the
LLaMA (Touvron et al., 2023) decoder-transformer of Vicuna. Following a
parameter-efficient LoRA fine-tuning of the Vicuna checkpoint on Flan-mini, we
obtain Flacuna. As expected, Flacuna outperforms Vicuna by a substantial
margin on most benchmark datasets, especially for reasoning-intensive tasks.
However, the performance of Flacuna still remains below Flan-T5 on the same
reasoning benchmarks. This could be attributed to the 15-times smaller dataset
of the instruction dataset which may contain less diverse samples.
Furthermore, full fine-tuning of Vicuna may narrow the gap with Flan-T5.

This work overall has the following contributions:

  1. 1.

Improving the problem-solving capability of Vicuna through parameter efficient
fine-tuning on Flan-mini.

  2. 2.

Introducing an instruction tuning dataset, Flan-mini, comprising a diverse set
of tasks and templates.

##  2 Training Details

#### Preparing the Flan-mini Collection.

Given the enormous size of the Flan Collection (Longpre et al., 2023), we
opted to work with a carefully selected subset that maintains a high level of
task diversity while reducing the overall dataset size. In Table 1, we present
the specific tasks included in our subset of Flan, along with their respective
dataset sizes. As the public release of the Flan Collection does not include
programming tasks, we augment the collection with existing code datasets.
Specifically, we include CodeContests (Li et al., 2022a), APPS (Hendrycks et
al., 2021a) and CodeSearchNet (Husain et al., 2019a). Following the data
processing pipeline of Flan Collection, we sample a fixed number of examples
from each dataset, where each example is randomly augmented with different
prompt templates. Specifically, the examples are processed with a pool of
handcrafted prompt templates and may be used as zero-shot examples or grouped
together with few-shot demonstrations (Longpre et al., 2023).

Dataset Name | Source | Dataset Size
---|---|---
Flan2021 | Flan | 388K
Public Pool of Prompts | Flan | 320K
Natural instructions v2 | Flan | 200K
CoT | Flan | 100K
Code Search | Husain et al. (2019b) | 100K
Code Contest | Li et al. (2022b) | 50K
Apps | Hendrycks et al. (2021b) | 50K
GPT4-Alpaca | GPT-4 | 52K
Code-Alpaca | ChatGPT | 20K
ShareGPT | ChatGPT | 60K
Total | - | 1.34M
Table 1: The Flan-mini Collection, used to train Flacuna.

#### Maintaining Vicuna’s Chatting Ability.

Vicuna has demonstrated remarkable chatting ability, achieving 90% of the
performance of ChatGPT. This indicates its significant potential as an open-
source alternative to closed-source large language models (LLMs) like ChatGPT.
To ensure that Flacuna retains Vicuna’s learned knowledge and chatting
ability, we incorporated various ChatGPT datasets, including Alpaca (Taori et
al., 2023), Code Alpaca (Chaudhary, 2023), and ShareGPT (Chiang et al., 2023),
into our Flan collection. Among these three datasets, Vicuna was originally
fine-tuned using the ShareGPT dataset. The final collection was then used to
train Flacuna.

#### Architecture.

We employed LoRA in the Vicuna model for fine-tuning on the Flan-mini
collection. We inserted the low-rank adapters on all the query and value
projection layers, resulting in a total trainable parameter count of 6.55M,
which is only around 0.05% of the parameter count of the original 13B Vicuna
model. The maximum input sequence length was set to 1280, and efficient
training was facilitated by utilizing bf16 precision.

#### Hyperparameter Details.

Flacuna was trained on 4$\times$A6000 GPUs for 1 epoch. We use 16 gradient
accumulation steps with a per-device batch size of 2, resulting in a total
batch size of 128. We used 3000 warm-up steps and a learning rate of 2e-5.

##  3 Evaluation Tasks and Results

###  3.1 Problem Solving Evaluation

To assess the problem-solving prowess of instructed large language models
(LLMs), InstructEval employs a range of benchmarks encompassing real-world
exams that delve into diverse topics. These benchmarks encompass complex
instructions, arithmetic problems, programming challenges, and causal
reasoning tasks. In order to excel in these benchmarks, models need to exhibit
a profound understanding of the world, demonstrate multi-hop reasoning
capabilities, showcase creativity, and employ a plethora of other cognitive
skills.

#### World Knowledge.

The Massive Multitask Language Understanding (MMLU) benchmark, introduced in
the work by Hendrycks et al. (2021c), serves as an assessment tool to gauge
the problem-solving aptitude and world knowledge of language models across
various subjects. It offers evaluations in both zero-shot and few-shot
settings, presenting a more challenging and human-like evaluation scenario.
The MMLU benchmark encompasses a comprehensive range of 57 subjects spanning
STEM, humanities, social sciences, and other domains. The difficulty levels of
the tasks within the benchmark vary from elementary to advanced professional
levels, providing a comprehensive assessment of the model’s capabilities in
problem-solving and domain understanding.

#### Complex Instructions.

The subset known as BIG-Bench Hard (BBH) comprises 23 highly demanding tasks
carefully selected from the BIG-Bench benchmark (Srivastava et al., 2022) to
specifically target tasks that are considered to surpass the current
capabilities of language models (Suzgun et al., 2022). BBH presents models
with intricate instructions that require advanced skills in navigation,
logical deduction, and fallacy detection.

#### Comprehension and Arithmetic.

Discrete Reasoning Over Paragraphs (DROP) is a reading comprehension task with
a mathematical focus. It challenges systems to engage in discrete reasoning by
analyzing passages extracted from Wikipedia articles. In order to excel in the
DROP task, a system needs to adeptly navigate references within a question and
identify the appropriate sections of the provided passage. Additionally, the
system must demonstrate proficiency in performing discrete operations like
addition, counting, or sorting.

#### Programming.

HumanEval serves as a problem-solving benchmark specifically designed for
assessing the performance of large language models that are trained on code
(Chen et al., 2021). The benchmark comprises 164 unique programming problems,
encompassing areas such as language comprehension, algorithms, and basic
mathematics. Some of the problems included in HumanEval are similar in nature
to straightforward software interview questions. In the evaluation process,
models are assessed based on the functional correctness of the code programs
they generate, with the criteria for correctness determined by the given
docstrings. HumanEval provides a comprehensive evaluation framework for
assessing the problem-solving capabilities of language models in a code-
centric context.

#### Causality.

The Counterfactual Reasoning Assessment (CRASS) benchmark is a novel dataset
and evaluation tool developed specifically to assess the causal reasoning
abilities of large language models. By employing counterfactual scenarios,
CRASS tests the model’s capability to identify and select appropriate causal
explanations. This benchmark provides a unique and rigorous evaluation
framework to gauge the causal reasoning capabilities of language models.

###  3.2 Alignment to Human Values

Noting the importance of aligning LLMs to human values, InstructEval
incorporates the Helpful, Honest, and Harmless (HHH) benchmark (Askell et al.,
2021). The benchmark showcases engaging dialogues between humans and
conversational assistants, challenging the model to discern and provide the
most appropriate response. It encompasses a diverse array of 61 honesty-
related, 59 helpfulness-related, and 58 harmlessness-related samples, along
with 43 unique instances falling within the "other" category. The inclusion of
the "other" category accounts for examples that embody values not explicitly
covered by honesty, helpfulness, or harmlessness.

###  3.3 Writing Experiments

For the writing experiment, we utilized the IMPACT dataset, which is readily
available in InstructEval. This comprehensive dataset consists of 50 prompts
across distinct categories, namely informative, professional, argumentative,
and creative. Following that, ChatGPT was assigned the responsibility of
scoring the models’ responses in terms of relevance (Rel.) and coherence
(Coh.) on a scale of 1 to 5. For more comprehensive information regarding this
evaluation, we refer readers to Chia et al. (2023).

###  3.4 Results

#### Comparative Baselines.

As baselines, we selected Vicuna (Zheng et al., 2023) and
StableVicuna111https://huggingface.co/CarperAI/stable-vicuna-13b-delta.

#### Few-shot Problem-solving.

We present the results of Flacuna on five datasets (see Table 2) from the
InstructEval benchmark, focusing on problem-solving tasks. In 4 out of 5
tasks, Flacuna outperformed Vicuna, showing an average performance improvement
of 5.6 points over the LLaMA backbone. However, it performed slightly worse on
code-related problem-solving tasks in the HumanEval dataset, with a margin of
0.6 points. Overall, the improvement in Flacuna compared to Vicuna is 5.1
points averaged over the five tasks.

Model | Size | MMLU (5-shot) | BBH (3-shot) | DROP$\star$ (3-shot) | CRASS (3-shot) | HumanEval (0-shot) | Avg.
---|---|---|---|---|---|---|---
Perf. | $\Delta$ | Perf. | $\Delta$ | Perf. | $\Delta$ | Perf. | $\Delta$ | Perf. | $\Delta$ | Perf. | $\Delta$
GPT-4 | - | 86.4 | - | - | - | 80.9 | - | - | - | 67.0 | - | - | -
ChatGPT | - | 70.0 | - | 49.5 | - | 64.1 | - | 90.5 | - | 48.1 | - | 64.5 | -
Flan-UL2 | 20B | 55.0 | - | 44.7 | - | 64.3 | - | 94.2 | - | 0.0 | - | 51.6 | -
Alpaca-Lora | 30B | 58.4 | +0.6 | 41.3 | +2.0 | 45.1 | -0.3 | 79.2 | +10.6 | 18.9 | +4.9 | 48.6 | +3.6
OpenAssistant | 30B | 56.9 | -0.9 | 39.2 | -0.1 | 46.0 | +0.6 | 67.2 | +1.4 | 23.1 | +9.1 | 46.5 | +1.5
OPT-IML | 30B | 38.6 | +11.3 | 31.3 | +3.0 | 47.5 | +28.0 | 67.2 | +32.5 | 9.1 | +7.9 | 38.7 | +16.5
Flan-T5 | 11B | 54.5 | +29.3 | 43.9 | +13.6 | 67.2 | +49.7 | 88.3 | +54.7 | 0.0 | +0.0 | 50.8 | +29.5
Flan-Alpaca | 11B | 50.9 | +25.7 | 23.3 | -7.0 | 62.3 | +44.8 | 90.2 | +56.6 | 0.0 | +0.0 | 45.3 | +24.0
Dolly V2 | 12B | 25.6 | -1.3 | 29.7 | +0.2 | 16.6 | -0.5 | 35.8 | +1.1 | 8.5 | -0.6 | 23.2 | -0.7
Flan-T5 | 3B | 49.2 | +25.9 | 40.2 | +15.9 | 56.3 | +43.7 | 91.2 | +60.2 | 0.0 | +0.0 | 47.4 | +29.2
ChatGLM | 6B | 36.1 | - | 31.3 | - | 44.2 | - | 51.1 | - | 3.1 | - | 33.2 | -
Mosaic-Chat | 7B | 37.1 | +1.9 | 32.0 | +1.1 | 20.2 | -7.4 | 47.5 | +13.6 | 17.7 | +7.4 | 30.9 | +3.3
StableVicuna | 13B | 49.2 | +3.0 | 37.5 | +0.4 | 34.3 | -1.0 | 67.5 | +8.7 | 15.9 | +2.5 | 40.9 | +2.7
Vicuna | 13B | 50.6 | +4.5 | 37.6 | +0.5 | 32.6 | -3.0 | 60.9 | +2.1 | 11.6 | -1.8 | 38.7 | +0.5
Flacuna | 13B | 51.1 | +5.0 | 39.3 | +2.2 | 43.6 | +8.0 | 74.1 | +15.3 | 11.0 | -2.4 | 43.8 | +5.6

Table 2: Evaluation results for problem-solving benchmarks. We denote the original performance across the benchmarks as Perf., while $\Delta$ denotes the change in performance compared to the corresponding foundation LLMs. $\star$ indicates that DROP is a held-in dataset. Model | Size | MMLU (0-shot) | BBH (0-shot) | CRASS (0-shot)
---|---|---|---|---
Flan-UL2 | 20B | 54.4 | 34.9 | -
OpenAssistant | 30B | 52.0 | 33.4 | -
OPT IML | 30B | 41.3 | 17.4 | -
TK-Instruct | 11B | 39.4 | 17.1 | -
Flan-T5-XXL | 11B | 54.1 | 39.5 | -
Dolly V2 | 12B | 25.4 | 22.3 | -
StableVicuna | 13B | 47.5 | 18.5 | 64.2
Vicuna | 13B | 48.3 | 28.3 | 65.7
Flacuna | 13B | 49.4 | 32.5 | 67.9
Table 3: 0-shot problem-solving evaluation of Flacuna and other baseline
models.

Out of the five problem-solving datasets, one of them, DROP, is categorized as
a held-in dataset. It is a part of our Flan collection and was utilized for
training Flacuna. As a result, we observed a significant performance boost of
11 points compared to Vicuna. The remaining datasets are considered held out.

#### 0-shot Problem-solving.

We conducted a 0-shot performance evaluation of Flacuna and compared it
against both Vicuna and StableVicuna. The results presented in Table 3
demonstrate a noteworthy performance leap by Flacuna compared to its
competitors. This improvement can be attributed to the training of Flacuna on
the high-quality Flan instruction dataset.

#### HHH Evaluation.

We conducted a further evaluation using BBH’s HHH evaluation dataset (see
Table 4), where Flacuna exhibited an impressive 11% improvement over Vicuna.
Notably, our instruction dataset collection aimed to enhance Vicuna’s problem-
solving abilities, but it also had a positive impact on its HHH performance.
This observation aligns with the experience of Flan-T5, which achieved a 24.2%
performance improvement over its T5 backbone after fine-tuning on Flan.

Model | Size | Harmlessness | Helpfulness | Honesty | Other | Avg. |  $\Delta$ Avg.
---|---|---|---|---|---|---|---
ChatGPT | - | 90.7 | 91.2 | 78.1 | 86.3 | 86.6 | -
Flan-Alpaca | 11B | 74.2 | 81.4 | 77.4 | 83.4 | 79.1 | +26.6
Flan-T5 | 11B | 75.9 | 75.3 | 75.1 | 79.6 | 76.7 | +24.2
Tk-Instruct | 11B | 70.1 | 54.8 | 62.3 | 76.0 | 65.8 | +13.3
T5 | 11B | 46.4 | 54.8 | 58.1 | 50.7 | 52.5 | -
Alpaca | 13B | 49.7 | 51.2 | 51.8 | 45.5 | 49.5 | -12.3
LLaMA | 13B | 57.2 | 61.0 | 57.0 | 72.0 | 61.8 | -
Dolly V2 | 12B | 51.7 | 59.9 | 47.0 | 58.1 | 54.2 | +9.1
Pythia | 12B | 41.3 | 46.1 | 43.6 | 49.3 | 45.1 | -
StableVicuna | 13B | 61.7 | 67.2 | 57.1 | 79.1 | 66.3 | +4.5
Vicuna | 13B | 62.0 | 66.1 | 52.4 | 74.4 | 63.7 | +1.9
Flacuna | 13B | 72.4 | 71.2 | 70.5 | 83.7 | 74.5 | +12.6
Table 4: Evaluation results for alignment to human values on the honesty,
helpfulness, and harmlessness (HHH) benchmark. Avg. denotes the average
performance, while $\Delta$ Avg. denotes the average improvement compared to
the corresponding foundation model.

Model | Size | Informative | Professional | Argumentative | Creative | Avg.
---|---|---|---|---|---|---
Rel. | Coh. | Rel. | Coh. | Rel. | Coh. | Rel. | Coh. | Rel. | Coh.
ChatGPT | - | 3.34 | 3.98 | 3.88 | 3.96 | 3.96 | 3.82 | 3.92 | 3.94 | 3.78 | 3.93
Flan-Alpaca | 11B | 3.56 | 3.46 | 3.54 | 3.70 | 3.22 | 3.28 | 3.70 | 3.40 | 3.51 | 3.46
Flan-T5 | 11B | 2.64 | 3.24 | 2.62 | 3.22 | 2.54 | 3.40 | 2.50 | 2.72 | 2.58 | 3.15
Dolly-V2 | 12B | 3.54 | 3.64 | 2.96 | 3.74 | 3.66 | 3.20 | 3.02 | 3.18 | 3.30 | 3.44
StableVicuna | 13B | 3.54 | 3.64 | 2.96 | 3.74 | 3.30 | 3.20 | 3.02 | 3.18 | 3.21 | 3.44
Vicuna | 13B | 3.60 | 3.96 | 3.74 | 3.82 | 3.82 | 3.56 | 3.82 | 3.92 | 3.75 | 3.82
Flacuna | 13B | 3.02 | 3.42 | 3.48 | 3.52 | 3.38 | 3.02 | 3.92 | 3.80 | 3.45 | 3.44

Table 5: Evaluation results for writing-based tasks.

#### Writing Evaluation.

While Flacuna primarily excels in problem-solving tasks, we made efforts to
maintain the impressive writing and chatting ability of Vicuna. To achieve
this, we incorporated conversational datasets generated by GPT-4, such as
GPT-4-Alpaca and ShareGPT, into the Flan-mini collection. However, despite
these efforts, we observed certain issues in Flacuna’s writing performance. In
some cases, it generates code snippets in response to prompts that are
unrelated to coding. We attribute this behavior to the significant data
imbalance, where the conversational dataset constitutes only 8.2% of the
entire data mixture. Prompt engineering techniques can help rectify such
issues.

We discovered that Flacuna generates responses of reasonable quality when
provided with the following template: ‘‘‘A chat between a curious user and an
artificial intelligence assistant. The assistant gives helpful, detailed, and
polite answers to the user’s questions. USER: definition of the task./n/n
question/n Output: ASSISTANT: ’’’. This finding suggests that with the
appropriate prompts, we can improve Flacuna’s chatting performance.

However, upon careful examination of the generated samples, it becomes
apparent that Flacuna does not outperform Vicuna as a writing assistant. This
observation is reinforced by the evaluation of the generated responses to the
prompts in the IMPACT dataset using ChatGPT, as depicted in Table 5. ChatGPT
consistently ranks Vicuna’s responses significantly higher than those of
Flacuna.

The subpar performance of Flacuna in writing-based scenarios can be attributed
to several factors. Firstly, the disproportionate scarcity of conversational
datasets in Flan may have contributed to this outcome. Additionally,
parameter-efficient tuning methods such as LoRA may limit the effectiveness of
the model in learning both problem-solving and general writing abilities.
Hence, we may explore other efficient training methods for LLMs in the future
(Lv et al., 2023).

An example of the prompt and Flacuna’s response is shown below.

Prompt: A chat between a curious user and an artificial intelligence
assistant. The assistant gives helpful, detailed, and polite answers to the
user’s questions. USER: You are tasked to demonstrate your writing skills in
professional or work settings for the following question. Write a user manual
for a household appliance, providing step-by-step instructions for its
operation, maintenance, and troubleshooting, using clear and simple language.
Output: ASSISTANT:
---
Flacuna’s response:
1\. Introduction Welcome to the user manual for the XYZ household appliance.
This appliance is designed to make your life easier and more convenient. With
this appliance, you can prepare your meals quickly and efficiently. 2\.
Operation To operate the appliance, follow these steps: a. Turn on the
appliance by pressing the power button. b. Select the desired cooking mode by
using the mode selector. c. Place the food inside the appliance. d. Close the
lid and press the start button. e. Wait for the appliance to complete the
cooking process. 3\. Maintenance To keep your appliance in good working
condition, follow these steps: a. Clean the appliance after each use. b. Check
the appliance for any signs of damage or wear and tear. c. If you notice any
issues, contact the manufacturer or a certified repair technician. 4\.
Troubleshooting If you encounter any issues with your appliance, try the
following troubleshooting steps: a. Check the power cord and plug to ensure
they are securely connected. b. Check the fuse to ensure it is not blown. c.
Check the appliance manual for troubleshooting tips. d. If the issue persists,
contact the manufacturer or a certified repair technician. 5\. Safety
Precautions Always follow these safety precautions when using the appliance:
a. Keep the appliance clean and free of debris. b. Do not use the appliance if
it is damaged or showing signs of wear and tear. c. Do not leave the appliance
unattended while it is in operation. d. Always use the appliance according to
the manufacturer’s instructions. 6\. Conclusion Congratulations on your new
XYZ household appliance! With proper use and maintenance, your appliance will
provide you with years of reliable service. If you have any questions or
concerns, please refer to the appliance manual or contact the manufacturer.

##  4 Limitations and Future Work

Despite the promising advancements of Flacuna compared to Vicuna, we have
identified some issues that require addressing:

  * •

If Flacuna is asked to provide descriptive answers to questions like “Present
arguments for or against lowering the age bar for drinking,” Flacuna generates
code snippets instead. This behavior could be attributed to its imperfect
understanding of instructions or a tendency to hallucinate.

  * •

Flacuna is still significantly behind Flan-T5 in terms of problem-solving
abilities.

  * •

Surprisingly, Flacuna exhibits inferior performance compared to both LLaMA and
Vicuna on coding-related problems. This outcome is unexpected, considering
that we incorporated numerous coding problem-solving datasets into our
instruction tuning collection.

  * •

Flacuna is trained with a maximum input sequence length of 1280 which limits
its ability to comprehend longer input sequences.

To address these limitations and known issues, we can explore the following
steps:

  * •

Based on previous studies, it has been observed that LoRA performs better with
larger models (Chia et al., 2023), such as those with 30B or 65B parameters,
and excels in task-specific settings. Therefore, in future work, we could
enhance Flacuna by fully fine-tuning Vicuna, without LoRA, particularly on the
Flan collection. Another future work is to train Flacuna on longer token
length.

  * •

We can incorporate the original Flan collection into the training process, as
it is fifteen times larger than the instruction dataset we used in this study.
Flan-T5 underwent training on this extensive collection, which resulted in
remarkable problem-solving performance.

  * •

The chatting or writing performance of Flacuna could be improved by
incorporating larger conversational datasets in Flan-mini and subsequently
training Flacuna on it.

## References

  * Chia et al. (2023) Yew Ken Chia, Pengfei Hong, Lidong Bing, and Soujanya Poria.  Instructeval: Towards holistic evaluation of instruction-tuned large language models, 2023.
  * Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.  Stanford alpaca: An instruction-following llama model, 2023.  URL https://github.com/tatsu-lab/stanford_alpaca.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.  URL https://vicuna.lmsys.org.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aur’elien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  Llama: Open and efficient foundation language models.  _ArXiv_ , abs/2302.13971, 2023.
  * Longpre et al. (2023) Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al.  The flan collection: Designing data and methods for effective instruction tuning.  _arXiv preprint arXiv:2301.13688_ , 2023.
  * Hu et al. (2021) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  _arXiv preprint arXiv:2106.09685_ , 2021.
  * Li et al. (2022a) Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Ré mi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals.  Competition-level code generation with AlphaCode.  _Science_ , 378(6624):1092–1097, dec 2022a.  doi: 10.1126/science.abq1158.  URL https://doi.org/10.1126%2Fscience.abq1158.
  * Hendrycks et al. (2021a) Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Xiaodong Song, and Jacob Steinhardt.  Measuring coding challenge competence with apps.  _ArXiv_ , abs/2105.09938, 2021a.
  * Husain et al. (2019a) Hamel Husain, Hongqi Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt.  Codesearchnet challenge: Evaluating the state of semantic code search.  _ArXiv_ , abs/1909.09436, 2019a.
  * Husain et al. (2019b) Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt.  CodeSearchNet challenge: Evaluating the state of semantic code search.  _arXiv preprint arXiv:1909.09436_ , 2019b.
  * Li et al. (2022b) Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals.  Competition-level code generation with alphacode.  _arXiv preprint arXiv:2203.07814_ , 2022b.
  * Hendrycks et al. (2021b) Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt.  Measuring coding challenge competence with apps.  _NeurIPS_ , 2021b.
  * Chaudhary (2023) Sahil Chaudhary.  Code alpaca: An instruction-following llama model for code generation.  https://github.com/sahil280114/codealpaca, 2023.
  * Hendrycks et al. (2021c) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.  Measuring massive multitask language understanding.  In _International Conference on Learning Representations_ , 2021c.  URL https://openreview.net/forum?id=d7KBjmI3GmQ.
  * Srivastava et al. (2022) Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. Iyer, Anders Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, Andrew Dai, Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakaş, B. Ryan Roberts, Bao Sheng Loe, Barret Zoph, Bartłomiej Bojanowski, Batuhan Özyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekci, Bill Yuchen Lin, Blake Howald, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta, César Ferri Ramírez, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison-Burch, Chris Waites, Christian Voigt, Christopher D. Manning, Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro, Colin Raffel, Courtney Ashcraft, Cristina Garbacea, Damien Sileo, Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Moseguí González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong-Ho Lee, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, Ellie Pavlick, Emanuele Rodola, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang, Ethan A. Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fatemeh Siar, Fernando Martínez-Plumed, Francesca Happé, Francois Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambattista Parascandolo, Giorgio Mariani, Gloria Wang, Gonzalo Jaimovitch-López, Gregor Betz, Guy Gur-Ari, Hana Galijasevic, Hannah Kim, Hannah Rashkin, Hannaneh Hajishirzi, Harsh Mehta, Hayden Bogar, Henry Shevlin, Hinrich Schütze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, Ian Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jaehoon Lee, Jaime Fernández Fisac, James B. Simon, James Koppel, James Zheng, James Zou, Jan Kocoń, Jana Thompson, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang, Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez-Orallo, Joseph Boudeman, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh D. Dhole, Kevin Gimpel, Kevin Omondi, Kory Mathewson, Kristen Chiafullo, Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin, Lidia Contreras-Ochando, Louis-Philippe Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros Colón, Luke Metz, Lütfi Kerem Şenel, Maarten Bosma, Maarten Sap, Maartje ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Ramírez Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew L. Leavitt, Matthias Hagen, Mátyás Schubert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael A. Yee, Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Starritt, Michael Strube, Michał Swędrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mo Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng, Nathan Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha S. Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah Alipoormolabashi, Peiyuan Liao, Percy Liang, Peter Chang, Peter Eckersley, Phu Mon Htut, Pinyu Hwang, Piotr Miłkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramón Risco Delgado, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan LeBras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Ruslan Salakhutdinov, Ryan Chi, Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang, Sahib Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bowman, Samuel S. Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima, Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee, Spencer Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen Prasad, Steven T. Piantadosi, Stuart M. Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsu Hashimoto, Te-Lin Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, Timofei Kornev, Timothy Telleen-Lawton, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj, Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Ramasesh, Vinay Uday Prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saunders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang, Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie J. Wang, Zirui Wang, and Ziyi Wu.  Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, 2022.
  * Suzgun et al. (2022) Mirac Suzgun, Nathan Scales, Nathanael Scharli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed Huai hsin Chi, Denny Zhou, and Jason Wei.  Challenging big-bench tasks and whether chain-of-thought can solve them.  _ArXiv_ , abs/2210.09261, 2022.
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harrison Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, David W. Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William H. Guss, Alex Nichol, Igor Babuschkin, S. Arun Balaji, Shantanu Jain, Andrew Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew M. Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba.  Evaluating large language models trained on code.  _ArXiv_ , abs/2107.03374, 2021.
  * Askell et al. (2021) Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan.  A general language assistant as a laboratory for alignment, 2021.
  * Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica.  Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
  * Lv et al. (2023) Kai Lv, Yuqing Yang, Tengxiao Liu, Qinghui Gao, Qipeng Guo, and Xipeng Qiu.  Full parameter fine-tuning for large language models with limited resources, 2023.