# Inferring the Phylogeny of Large Language Models
and Predicting their Performances in Benchmarks

Nicolas Yax
LNC2, INSERM, Paris, France
DEC, ENS, Paris, France
INRIA Bordeaux-Sud-Ouest, France
<EMAIL_ADDRESS>Oudeyer∗
INRIA Bordeaux-Sud-Ouest, France
&Stefano Palminteri∗
LNC2, INSERM, Paris, France
DEC, ENS, Paris, France

###### Abstract

This paper introduces PhyloLM, a method applying phylogenetic algorithms to
Large Language Models to explore their finetuning relationships, and predict
their performance characteristics. By leveraging the phylogenetic distance
metric, we construct dendrograms, which satisfactorily capture distinct LLM
families (across a set of 77 open-source and 22 closed models). Furthermore,
phylogenetic distance predicts performances in benchmarks (we test MMLU and
ARC), thus enabling a time and cost-effective estimation of LLM capabilities.
The approach translates genetic concepts to machine learning, offering tools
to infer LLM development, relationships, and capabilities, even in the absence
of transparent training information.

Inferring the Phylogeny of Large Language Models
and Predicting their Performances in Benchmarks

Nicolas Yax LNC2, INSERM, Paris, France DEC, ENS, Paris, France INRIA
Bordeaux-Sud-Ouest, France<EMAIL_ADDRESS>Pierre-Yves Oudeyer∗ INRIA
Bordeaux-Sud-Ouest, France Stefano Palminteri∗ LNC2, INSERM, Paris, France
DEC, ENS, Paris, France

##  1 Introduction

††* equal contribution

The language model landscape is vast and rapidly expanding, comprising both
private and open-access models. Evaluating these models presents challenges
due to the sheer volume and the complexity of assessing their true
capabilities. The evaluation methods used today mostly rely on a multitude of
benchmarks, each focused on specific domains like reasoning or question-
answering Hendrycks et al. (2021) Chollet (2019) Srivastava et al. (2023). Yet
they face inherent limitations, including the fact that they are rather
specific Chang et al. (2023), potential test set contamination Deng et al.
(2023) and the high cost of running multiple benchmarks Liang et al. (2023).
Moreover, the opaque nature of training specifics and data adds complexity to
the understanding of progress in LLMs Liao and Vaughan (2023), making it
difficult to maintain an accurate record of advancements in the field.

Many models inherit features from existing ones, such as initial weights or
training data. Inferring and mapping these relationships could enhance our
understanding of model advancement, offering insights beyond traditional
benchmarking.

One can draw an analogy to population genetics, where algorithms enable to
reconstruct phylogenetic trees to understand evolutionary relationships among
species Takezaki and Nei (1996). These methods, initially developed for
biology, have already been applied to infer aspects of the evolution of
cultural artefacts such that tale evolution Tehrani and d’Huy (2017); d’Huy
(2013), languages Atkinson et al. (2008); Gray et al. (2010), or craft
assemblages Tehrani and Collard (2009), yielding useful insights. We reasoned
that a similar methodology can be applied to a recent kind of cultural
artefacts: language models. By mapping genetic concepts to LLMs, we let models
generate tokens in various context and compare them generating genetic
distance matrices (see Fig 2) and dendrograms, offering insights into the
genealogy and, ultimately, the skills of these models.

The potential of leveraging genetic frameworks for LLMs lies in understanding
how models evolve and diverge, shedding light on their interconnectedness and,
indirectly, their skillsets. This paper explores applying genetic concepts to
LLMs, tracing model genealogy through distance matrices, and extending the
genetic framework to reflect model skills.

Figure 1: Analogy between running human genetic studies and LLMs genetic
studies. The first stage consists in selecting genes (for both humans and
LLMs). Then alleles are collected for each individual in the population and
will be used to compare the populations (either populations of humans or LLMs
seen as populations). Finally these data go through the Nei distance
computation that returns a distance matrix that can then be turned into
dendrograms using the NJ algorithm in the same way for both humans and LLMs.
$D(P_{1},P_{2})=-\log(S(P_{1},P_{2}))\qquad S(P_{1},P_{2})=\frac{\sum_{g\in
G}\sum_{a\in A_{g}}P_{1}(a|g)P_{2}(a|g)}{\sqrt{(\sum_{g\in G}\sum_{a\in
A_{g}}P_{1}(a|g)^{2})(\sum_{g\in G}\sum_{a\in A_{g}}P_{2}(a|g)^{2})}}$ Figure
2: Nei score distance computation Takezaki and Nei (1996) with $P_{1}$ and
$P_{2}$ two populations seen as probability distribution of alleles given a
gene estimated empirically in the selected populations. $G$ is the set of
genes considered and $A_{G}$ the set of alleles for this gene. Matrix $S$ is
the similarity matrix (bounded in [0,1] - which is easier to plot) while $D$
is the distance matrix used to plot dendrograms.

This paper makes several contributions. Firstly, it introduces an algorithm,
PhyloLM, inspired by a simplified phylogenetic approach, tailored specifically
for Large Language Models (LLMs). Secondly, it delves into exploring the
hyperparameter space to strike a balance between precision and computational
efficiency. Thirdly, it investigates the correlations between phylogenetic
distance and the finetuning families of models (see Fig 1). This analysis
primarily focuses on open-access model families (Llama Touvron et al. (2023a,
b), Mistral Jiang et al. (2023), Bloom BigScienceWorkshop et al. (2023),
Pythia Biderman et al. (2023), Falcon Almazrouei et al. (2023), OPT Zhang et
al. (2022) and Qwen Bai et al. (2023) families), where ground truth
information is available, while also providing insights into finetuning
relationships for select proprietary models (GPT-3 Brown et al. (2020), 3.5
Ouyang et al. (2022) and 4 OpenAI et al. (2023) families). Lastly, it examines
the correlations between phylogenetic distance and model performance on
several benchmarks (MMLU Hendrycks et al. (2021) and ARC Chollet (2019) are
presented here; more benchmarks have been tested and not included here for
space reason).

##  2 Methods

###  2.1 How to apply phylogenetic algorithms to LLMs ?

Transposing this methodology to Large Language Models (LLMs) involves
establishing analogies for genes, alleles, and populations. In the current
landscape, LLMs predominantly operate on an autoregressive basis, wherein they
learn the conditional probability denoted as $LLM(t|c)$. Here, $LLM$
represents the probability learned by the language model, $t$ signifies a
token, and $c$ denotes the context. Drawing a parallel with the notation for
populations in the Nei genetic distance (see Fig 2) Takezaki and Nei (1996),
$P_{1}(a|g)$, one can draw an analogy suggesting that tokens align with
alleles, contexts can be likened to genes, and LLMs play the role of
populations in this context as we will now explain.

In the realm of genetics, populations are conceptualized as probability
distributions of DNA, represented by $P(a|g)$, where $a$ stands for alleles at
gene locations. This abstract construct can be leveraged to generate DNA by
sampling alleles for all required genes, akin to how Large Language Models
generate text with tokens in contexts, expressed as $LLM(t|c)$ ($t$ being the
token most likely to follow text $c$). Each of these texts constitutes an
thread of DNA, comprised of tokens (alleles) sampled in contexts (genes)
according to a probability distribution defined by the LLM.

To elucidate this crucial point, consider a tokenized text sequence: ’I’
’_like’ ’_choco’ ’late’. This sequence can be analogous to a DNA thread
represented as ’I_like_chocolate’. Breaking it down, the allele I corresponds
to the gene $\epsilon$ (empty text), _like aligns with the gene I, _choco
associates with the gene I_like, and late is linked to the gene I_like_choco.
Now, consider another individual represented by ’I’ ’_prefer’ ’_ice’ ’_cream’.
These two individuals share exactly two genes: $\epsilon$, for which they
possess the same allele I, and the gene I, for which they have distinct
alleles (_like and _prefer). They do not share any further genes, as their
prefixes diverge beyond this point.

Implementing the phylogenetic algorithm in the context of Large Language
Models involves the selection of specific genes111Which is not uncommon in
genetics as well. Evolutionary studies are generally about a series of well
selected genes and not the full genome Grünwald et al. (2017) for comparison
and to estimate the proportion of the population sharing alleles at a
particular gene, for each gene and for each LLM. Then the formula 2 is
employed to get the final distance matrix. For a given gene $g$ and allele
$a$, the proportion of the population with allele $a$ for gene $g$ corresponds
to $LLM(a|g)$.

The algorithm is illustrated in Figure 1. The initial step involves collecting
model outputs to genes (alleles). Given a set of LLMs, a set of genes (called
genome), and the specified number of individuals in each population as $N$,
the models are queried for a single token $N$ times. The set of possible
alleles is thus the vocabulary of the models. This process generates the
matrix $P$, which serves as an approximation of $P(a|g)$, the proportion of
the population with allele $a$ to gene $g$.

Subsequently, based on this approximation, the distance matrix $D$ is computed
using the Nei genetic distance formula Takezaki and Nei (1996) depicted in
equation 2.

###  2.2 How to choose a good genome ?

The algorithm’s essence lies in allowing Large Language Models (LLMs) to
complete contexts (genes) and assessing the differences in their completions
to estimate model distances. The resulting distance matrix reflects the
average differences between models on selected genes.

We turn to genetics to map models as we seek new tools to distinguish between
the influx of upcoming LLMs and existing ones systematically and
automatically. For effective differentiation, we require a genome for which
completions are not too different between models (else no allele will match
and the distance will be maximal between all models) but not too similar (else
all model will appear to be the same). The ideal is to go the middle way :
genes with moderate completion variance between models.

Genes made from classical training data can expose the genome to
contamination, making a very low entropy genome. As such, we used test sets
from recent benchmark as LLMs shouldn’t be trained on these. We used 2
datasets to study the impact of the nature of the genome on results : open-
web-math Paster et al. (2023) and MBXP Athiwaratkun et al. (2023) as both are
recent dataset, the first in reasoning and the second in coding, both trending
topics in LLMs refinement recently thus topics that are likely to yield much
variance between families of models.

As for extracting genes from benchmarks, we uniformly select lines from the
solution column in the dataset (as it is the one with the strongest narrative
and the longest when it comes to reasoning and coding) and truncate the text
randomly between the 20th and 100th character to leave it open for LLMs to
complete the sentence given a small context.

###  2.3 How hyperparameters choice affect the final distance matrix ?

We designed 2 complementary methods to estimate the right hyperparameters to
run the algorithm. First we investigate how $G$, the size of the genome, and
$N$, the number of probes for each gene, influence the variance of the
distance matrix. Then, once the variance is controlled, what combination of
$G$ and $N$ approximate a very high $G^{\prime}$ and $N^{\prime}$ distance
matrix.

To assess the impact of the number of genes $G$ and number of individuals $N$
for each dataset, the algorithm was executed across a range of genome sizes
(varying between 16 and 256 genes per run) and $N$ (ranging from 8 to 128)
building similarity matrices (the matrix $S$ in 2 \- rather than distance
matrices as they are easier to compare). This investigation process, aimed at
testing the highest possible values for hyperparameters, is particularly
computationally expensive. Therefore it was only run on the 5 smallest OPENAI
models (ada,babbage,text-ada-001,text-babbage-001 and babbage-002) in order to
minimize the costs. Thus distance matrices are $5\times 5$ making it only an
estimate of what could be a larger distance matrix.

To investigate the variance, for each value of $G$, 8 independent genomes
(with different genes) are composed, each genome is probed $N$ times giving 8
similarity matrix $S_{G,N,i}$ for genome $i\in[|0,8\|]$ of size $G$ and $N$
probes per gene. A variance computation over this set of matrix is finally
performed yielding a matrix V containing the variance of each distance between
2 models :
$V^{2}=\frac{1}{8}\sum_{i}\left(S_{G,N,i}-\left(\frac{1}{8}\sum_{j}{S_{G,N,j}}\right)\right)^{2}$.
The final variance score is the mean value of every coefficient in the
variance matrix $V$.

Then we investigated the impact of these hyperparameters when trying to
approximate a high precision matrix. For this purpose, we compute the variance
around a very expensive distance matrix $S_{G^{\prime},N^{\prime}}$ with
$G^{\prime}=2048$ and $N^{\prime}=128$. Genomes for the high precision matrix
are independent from the lower size genomes used to estimate it. The formula
to compute this variance around the high precision matrix is $V^{\prime
2}=\frac{1}{8}\sum_{i}\left(S_{G,N,i}-S_{G^{\prime},N^{\prime}}\right)^{2}$.
The final metric is the mean value in the matrix.

###  2.4 How to align different tokenizers ?

In scenarios where models do not share the same tokenizer, aligning alleles
can pose challenges. For instance, if the context is "The president of the US
is Joe," and one model completes with "Biden" while another completes with
"Bi," they might be incorrectly considered as different alleles. This
discrepancy arises because the second model’s tokenizer could have split the
same completion into two distinct tokens.

To mitigate this issue, a proxy approach was employed by using the first 4
characters of the generated text as a representation of the first token.
Practically, each model was instructed to generate at least 4 tokens and the
comparison focused on the first 4 characters in the concatenation of these
tokens.

###  2.5 Dendrograms

From a distance matrix obtained by the phylogenetic algorithm it is usual to
plot dendrograms representing a possible evolution between the entities in the
distance matrix. For this purpose many different algorithm exist and we chose
the Neighbour Joining (NJ) technique Saitou and Nei (1987) for its simplicity
and efficiency.

###  2.6 How to predict benchmark scores from genetic distance

A multi-layer perceptron (MLP) was trained on an Multidimensional scaling
(MDS) of the distance matrix to predict the MMLU and ARC accuracy of the
models. The MDS makes it possible to reduce the input dimension but also to
make the input space more relevant for regression tasks. The regressor takes
as input a 15 dimension MDS embedding of the model and tries to predict its
benchmark score while being trained on the embedding and scores of all the
other models in the study. An MSE loss between the output of the model and the
true performance on the benchmark was used with Adam optimizer initialized
with a learning rate of $10^{-3}$.

We used a 2 layer MLP as regressor with 200 neurons on the hidden layer and
finally 1 output neuron. The non linearity on the hidden layer is relu and the
final non linearity is sigmoid to scale the output between 0 and 1 (as the
benchmark score is between 0% and 100%).

##  3 Experiments and Results

###  3.1 What is the impact of hyperparameters on the distance matrix ?

We first ran the variance estimation process explained in the Methods section
and plotted the results in figure 3. These Figures shows a clear decrease in
the variance as the number of genes grows. More in details, one can notice
that the variance seems lower for lower values of $N$. This is due to the fact
that, with low $N$, very different models will have a particularly low chance
to generate the same allele to the same gene. Therefore the similarity matrix
will be really close to the identity matrix as $N$ tends to 0. If all 8
matrices are identity, the variance is very low but the results are not
satisfying.

As a result, we know $G$ monitors the variance of the matrix and $N$, from a
mathematical standpoint, estimates the probability of alleles for the
different genes. Thus, having a high $N$ means a better approximate of the LLM
true token distribution yielding more accurate result. $N$ acts as a precision
parameter. However the total cost of the algorithm is proportional to $G\times
N$ making it extremely expensive both to reduce the variance and to increase
the precision. As such, what would be a good couple of hyperparameters
($G$,$N$) that accurately approximates a particularly expensive low variance
and high precision matrix ?

As shown in Figure 3, the number of probes is very important to efficiently
estimate the high precision matrix. Indeed, for each value of number of
probes, the performance saturates from some $G$ value making less and less
improvement when $G$ increases. Thus this figures gives an optimal $G$ for a
given $N$ in order to approximate the high precision matrix efficiently. The
total cost of the algorithm in tokens being proportional to $G\times N$, we
found a good tradeoff between variance and precision was around $G=128$ and
$N=32$.

The estimated cost to run the algorithm per model is therefore
$128\,\mbox{genes}\times 32\,\mbox{probes}\approx 4096$ queries of $\approx
20$ tokens. As a point of reference, conducting the MMLU benchmark requires
around 14,000 queries on significantly longer prompts ($\approx 70$ tokens
each), making it an order of magnitude more expensive in terms of the number
of tokens required.

Figure 3: Hyperparameters impact on distance matrices Left shows the variance
of distance matrices for different number of genes and N in the math
benchmark. Each genome of specified size contains different and independant
genes from the other matrices for a total of 8 distance matrix for each data
point in the figure. Right shows the variance computed around a distance
matrix made of 2048 genes and N=128 in the math benchmark.

Figure 4: Phylogenetic tree reconstruction. Left is the ground truth and right
is the reconstruction from the phylogenetic algorithm on the math genome. Only
the leaves have been included in the reconstruction as phylogenetics algorithm
usually don’t include common ancestors.

Figure 5: Predictions from the MLP compared to ground truth for every model
(leave-one-out method). Left is prediction on MMLU and right is on ARC both
from the math genome. Each point is the predicted score of the model when the
regressor is trained on data from all other models but the given point. For
the sake of clarity, names of models have been omited and replace by the color
of models from the dendrogram plot.

###  3.2 Can we trace back the genealogy of LLMs using tools from genetics ?

There is a very important point to understand when visualizing dendrograms of
LLMs. In genetics, all species studied are present living populations.
Therefore, branches in the tree show potential speciations that occured a
while ago and current species are a result of all this evolutionary process
and, therefore, are leaves of the tree. In LLMs, common ancestors are in the
studied populations, indeed Mistral 7B is the common ancestor of Mistral 7B
Instruct and Zephyr 7B Alpha. However this algorithm will put all models at
the leaves while in fact Mistral should be at a speciation node ! Therefore,
without the assumption that no ’common ancestor’ model are in the study, the
dendrogram has to be seen as clustering algorithm based on the distance
matrix.

Another assumption in genetics is challenged when it comes to LLMs : genetic
drift. Indeed, in genetics, after speciation, populations are assumed to
evolve independently leading to an increase in their genetic differences over
time which justifies the representation in an evolutionary trees. The only
reason populations could become closer and closer after a speciation is
environmental pressure : if there is a need to fly in the environment, many
species may develop wings over time. In LLMs, finetuning models is equivalent
to applying an environmental pressure on them to make them evolve in the
desired direction. Thus finetuning LLMs on the same dataset or on the same
topic (instruction following, math skills, …) may alter their DNA in a similar
manner. Where this is quite rare in genetics it is extremely common in LLMs.
We’ll see in practice how this affects the phylogenetic trees when this
assumption is broken.

Figure 6: Inferred phylogenetic tree of LLMs on the math genome.

Let’s first simplify the problem and take a tree from the Mistral family for
which we know the genealogy Arc53 (2023); mlabonne (2023a); Vallego (2024);
Ullah (2024); Tenyx (2024) (see Fig 4). We ran the NJ algorithm only including
the leaves models to align with the genetics assumptions, the results validate
the method as we perfectly get back the ground truth phylogenetic tree.

On the other hand, as the algorithm was designed to operate in a black box
setting without additional information, we cannot verify this assumption in
general. The dendrogram including most LLMs, computed from the distance matrix
can be found in Figure 6 and has to be read as a visualisation of the distance
matrix (shown in appendix - Fig 8).

At first sight, models are grouped by families showing the same clusters as in
the similarity matrix. Looking more in details in the tree reveals more
intricate structures within the families that might be challenging to
visualize in the similarity matrix plot.

More precisely, the shades of blue cluster comprises all Llama model with a
different shade for each version of LLama (different versions and number of
parameters) showing a fine grained classification of model beyond the general
family of models.

Evolution can make models close in 2 different manners : either the models
have the same parents but are finetuned on 2 different datasets (siblings) or
they have different parents but are finetuned in a similar manner (can be seen
as environmental pressure that makes species develop similar traits). This
effect is visible in the TigerBot Llama cluster showing two branches from
different version of llama trained on a similar dataset Chen et al. (2023a).
On the other hand, the Bloom based TigerBot brand stays quite far from the
llama one.

Another interesting cluster when comparing both genomes is probably the OPT /
GPT-3 / Falcon part of the tree. Indeed, OPT, GPT-3 and FalconRW 1B are mixed.
Seeing OPT mixed with FalconRW is not surprising as both where trained on part
of the Common Crawl dataset but seeing GPT-3 as well might indicate OPENAI
trained these models on the dataset as well (which we don’t know). On the
other hand, we see small variations in the code genome (shown in appendix -
Fig 10) : GPT-3 is mixed with FalconRW 1B and OPT is clustered on its own with
FalconRW 7B and even TigerBot 7B Base (based on Bloom), a neighbour cluster of
OPT in the code genome similarity matrix.

Lastly, the only cluster for which we don’t perfectly know the genealogy : GPT
models. Clearly the older GPT models (GPT-3 - see Tab 1 in appendix) form
their own clusters sometimes mixed with open access models. However, the
longest branch in both genome contains the most recent openai models including
ChatGPT and GPT-4 showing a strong link between these models as well as the
fact that, in terms of closeness, no model in this study approximates them
efficiently. This branch additionally splits gpt-3.5 and gpt-4 models by their
families but also by their release date.

###  3.3 Can we infer model capabilities from the genetic distance ?

The results shown in Figure 5 indicate that the accuracy of most models can be
predicted essentially from the genetic distance to other models for which the
performance is known for the MMLU and ARC benchmarks. This observation
underscores that the distance matrix not only clusters families of models but
also captures manifold features that reflect skills of the models, including
their distinct abilities and performance on various benchmarks. The predictive
power of the genetic distance highlights the utility of the genetic algorithm
in discerning nuanced variations among different models.

###  3.4 How robust are similarity matrices across different nature of genomes
?

In dendrograms and benchmark prediction, both genomes returned fairly similar
hyperparameters evolution (see Fig 3 and 11), similar similarity matrices (see
Fig 8 and 9), similar clusters in dendrograms (see Fig 6 and 10) and finally
close benchmark prediction performances (see Fig 5 and 13). This shows a
general robustness of the results to the nature of the genes used. However,
some details differ, like the relative positioning of OPT / GPT-3 and Falcon
models that seem to be the major difference between these models probably
because they were trained on different code datasets. Moreover, this study
only includes 2 genomes which is not enough to generalise to a larger scale.
Further studies should be conducted to strengthen this point.

##  4 Discussion

Here we show that an algorithm, inspired by those used in genetic analysis, is
successful in reconstructing important aspects of the history of models based
solely on their outputs to diverse short queries. By leveraging the genetic
distance matrix, it becomes feasible to robustly trace the relationships and
evolution of models over time. This is particularly evident in the constructed
dendrograms, where clear clusters align with distinct families of Large
Language Models, offering a visual representation of their evolutionary
trajectories or at least their training similarity.

Furthermore, the genetic distance, derived from the algorithm, is not limited
to capturing only training relationships but extends its utility to
correlating with the performances of models on MMLU and ARC. The observation
that an MLP trained on the genetic distance matrix can accurately predict MMLU
accuracy underscores the nuanced relationship between the genetic distances
and the models’ capabilities. This discovery has the potential to also provide
additional metrics for domains where benchmarking isn’t possible (artistic
capabilities, creativity, …).

Finally, we would like to emphasize the applicability of these methods to
proprietary models. Understanding the finetuning relationships and performance
characteristics of private models is often challenging due to limited access
to training details and data. However, PhyloLM offers a valuable tool for
gaining insights into these aspects, allowing for a more nuanced understanding
of their evolutionary paths and capabilities. Overall, these methods
contribute to a robust and insightful analysis of the history, relationships,
and performance of Large Language Models, even in cases where detailed
training information is not publicly available.

##  5 Acknowledgments

This work was granted access to the HPC/IA resources of [IDRIS HPE Jean Zay
A100] under the allocation 2023- [AD011013693R1] made by GENCI. SP is
supported by the European Research Council under the European Union’s Horizon
2020 research and innovation program (ERC) (RaReMem: 101043804), the Agence
National de la Recherche (CogFinAgent: ANR-21-CE23-0002-02; RELATIVE:
ANR-21-CE37-0008- 01; RANGE: ANR-21-CE28-0024-01), the Alexander Von Humboldt
foundation and a Google unrestricted gift.

##  6 Ethics and limitations

This work focuses on providing tools to facilitate the understanding of the
genealogy of language models, their relationships and their performances. As
such, we don’t believe PhyloLM could cause ethical concerns on the insights it
provides about LLMs nor about its usage.

This paper introduces this new algorithm and tries to better understand how
genetics can be used in the LLM framework, as such we first introduced the
general analogy and then mapped it into a more precise algorithm using very
general tools designed for genetics. We do not insinuate it is the only way to
apply genetics concepts on LLMs and encourage researchers to try other ideas
in this regard. In fact we believe these tools should be updated to better fit
the LLM framework.

Then we tested a range of hyperparameters on a 5x5 distance matrix to estimate
good hyperparameters. We didn’t investigate how noisy these results are
knowing we only used 5 models on 2 genomes due to the very high computational
cost. Indeed these 5 models are of a relatively low size and all from the same
company which may not reflect very well all the variability of LLMs in the
field. Therefore the hyperparameters we propose might be very well fitted for
reasoning / coding abilities on the GPT lower size models but not very much on
the open access or even chat models or more creative skills.

Subsequently we showed that PhyloLM, with the hyperparameters we found and
only running on the leaves, finds back the ground truth phylogenetic tree for
both genomes in our case studies. We did not investigate how robust this tree
is. Indeed the NJ algorithm can be quite unstable with LLMs data and further
studies should be run to evaluate the robustness of these trees on this new
type of data.

Thereafter, the global phylogenetic trees show quite robust clustering across
the two genomes, with some differences on the code genome linked to their code
training data. However this result should be replicated on more genomes and
should also include more LLMs to be more significant and give a wider overview
of the stability of this algorithm across many different natures of genomes
and families of models. Indeed, sometimes clusters mix with each other like
GPT, OPT and Falcon, but it can be difficult to know why. Papers in the LLM
community are often rather obscure about their training data and, even if we
have been able to see that they share some training data, this does not
explain why FalconRW 1B is a lot closer to OPT and GPT-3 models than FalconRW
7B that is supposed to be trained on the same data. Either more transparency
is needed to understand if the algorithm found something untold in the LLM
papers or if it is a flaw in the method. This requires further investigations.

Lastly, predictions of performances in benchmarks attain good levels. However
we know the algorithm is able to predict the families on one side but also
that models from the same family tend to share loosely similar benchmark
performances. Even if we see a real variance in the prediction within families
(like the llama family) the question as to whether the model predicts the
score or just the family (that then correlates with the score) is to be
answered. Furthermore we didn’t investigate whether it is possible to predict
the performance of a model from a family only having data from a different
family.

## References

  * Almazrouei et al. (2023) Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023.  The falcon series of open language models.
  * Arc53 (2023) Arc53. 2023.  Arc53/docsgpt-7b-mistral.
  * Athiwaratkun et al. (2023) Ben Athiwaratkun et al. 2023.  Multi-lingual evaluation of code generation models.
  * Atkinson et al. (2008) Quentin D Atkinson, Andrew Meade, Chris Venditti, Simon J Greenhill, and Mark Pagel. 2008.  Languages evolve in punctuational bursts.  _Science_ , 319(5863):588–588.
  * Bai et al. (2023) Jinze Bai et al. 2023.  Qwen technical report.
  * Biderman et al. (2023) Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. 2023.  Pythia: A suite for analyzing large language models across training and scaling.
  * BigScienceWorkshop et al. (2023) BigScienceWorkshop et al. 2023.  Bloom: A 176b-parameter open-access multilingual language model.
  * Brown et al. (2020) Tom B. Brown et al. 2020.  Language models are few-shot learners.
  * CausalLM (2023a) CausalLM. 2023a.  Causallm/14b.
  * CausalLM (2023b) CausalLM. 2023b.  Causallm/7b.
  * Chang et al. (2023) Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. 2023.  A survey on evaluation of large language models.
  * Chen et al. (2023a) Ye Chen, Wei Cai, Liangmin Wu, Xiaowei Li, Zhanxuan Xin, and Cong Fu. 2023a.  Tigerbot: An open multilingual multitask llm.
  * Chen et al. (2023b) Zhihong Chen, Junying Chen, Hongbo Zhang, Feng Jiang, Guiming Chen, Fei Yu, Tiannan Wang, Juhao Liang, Chen Zhang, Zhiyi Zhang, Jianquan Li, Xiang Wan, Haizhou Li, and Benyou Wang. 2023b.  Llm zoo: democratizing chatgpt.  https://github.com/FreedomIntelligence/LLMZoo.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
  * Chollet (2019) François Chollet. 2019.  On the measure of intelligence.
  * Conover et al. (2023) Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023.  Free dolly: Introducing the world’s first truly open instruction-tuned llm.
  * Deng et al. (2023) Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. 2023.  Investigating data contamination in modern benchmarks for large language models.
  * d’Huy (2013) Julien d’Huy. 2013.  Polyphemus (aa. th. 1137): A phylogenetic reconstruction of a prehistoric tale.  _Nouvelle Mythologie Comparée/New Comparative Mythology_ , 1(1):http–nouvellemythologiecomparee.
  * Gray et al. (2010) Russell D Gray, David Bryant, and Simon J Greenhill. 2010.  On the shape and fabric of human history.  _Philosophical Transactions of the Royal Society B: Biological Sciences_ , 365(1559):3923–3933.
  * Grünwald et al. (2017) Niklaus Grünwald, Sydney Everhart, Brian Knaus, and Zhian Kamvar. 2017.  Best practices for population genetic analyses.  _Phytopathology_ , 107.
  * Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021.  Measuring massive multitask language understanding.
  * Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023.  Mistral 7b.
  * Köpf et al. (2023) Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. 2023.  Openassistant conversations – democratizing large language model alignment.
  * Liang et al. (2023) Percy Liang et al. 2023.  Holistic evaluation of language models.
  * Liao and Vaughan (2023) Q. Vera Liao and Jennifer Wortman Vaughan. 2023.  Ai transparency in the age of llms: A human-centered research roadmap.
  * Lv et al. (2023) Kaokao Lv, Wenxin Zhang, and Haihao Shen. 2023.  Supervised fine-tuning and direct preference optimization on intel gaudi2.
  * Mitra et al. (2023) Arindam Mitra, Luciano Del Corro, Shweti Mahajan, Andres Codas, Clarisse Simoes, Sahaj Agrawal, Xuxi Chen, Anastasia Razdaibiedina, Erik Jones, Kriti Aggarwal, Hamid Palangi, Guoqing Zheng, Corby Rosset, Hamed Khanpour, and Ahmed Awadallah. 2023.  Orca 2: Teaching small language models how to reason.
  * mlabonne (2023a) mlabonne. 2023a.  mlabonne/neuralhermes-2.5-mistral-7b.
  * mlabonne (2023b) mlabonne. 2023b.  mlabonne/neuralhermes-2.5-mistral-7b.
  * OpenAI et al. (2023) OpenAI et al. 2023.  Gpt-4 technical report.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.
  * Paster et al. (2023) Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. 2023.  Openwebmath: An open dataset of high-quality mathematical web text.
  * Saitou and Nei (1987) N Saitou and M Nei. 1987.  The neighbor-joining method: a new method for reconstructing phylogenetic trees. _Molecular Biology and Evolution_ , 4(4):406–425.
  * Srivastava et al. (2023) Aarohi Srivastava et al. 2023.  Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.
  * Takezaki and Nei (1996) Naoko Takezaki and Masatoshi Nei. 1996.  Genetic Distances and Reconstruction of Phylogenetic Trees From Microsatellite DNA.  _Genetics_ , 144(1):389–399.
  * Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca.
  * Tehrani and d’Huy (2017) Jamie Tehrani and Julien d’Huy. 2017.  _2017\. Phylogenetics Meets Folklore: Bioinformatics Approaches to the Study of International Folktales_, pages 91–114.
  * Tehrani and Collard (2009) Jamshid J Tehrani and Mark Collard. 2009.  On the relationship between interindividual cultural transmission and population-level cultural diversity: a case study of weaving in iranian tribal populations.  _Evolution and Human Behavior_ , 30(4):286–300.
  * teknium (2023a) teknium. 2023a.  teknium/openhermes-2-mistral-7b.
  * teknium (2023b) teknium. 2023b.  teknium/openhermes-2.5-mistral-7b.
  * teknium (2023c) teknium. 2023c.  teknium/openhermes-7b.
  * Tenyx (2024) Tenyx. 2024.  Tenyxchat: Language model alignment using tenyx fine-tuning.
  * Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a.  Llama: Open and efficient foundation language models.
  * Touvron et al. (2023b) Hugo Touvron et al. 2023b.  Llama 2: Open foundation and fine-tuned chat models.
  * Tunstall et al. (2023) Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. 2023.  Zephyr: Direct distillation of lm alignment.
  * Ullah (2024) Imran Ullah. 2024.  Imran1/medchat3.5.
  * Vallego (2024) Jorge Vallego. 2024.  neovalle/h4rmoniousanthea.
  * Wang et al. (2023) Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. 2023.  Openchat: Advancing open-source language models with mixed-quality data.  _arXiv preprint arXiv:2309.11235_.
  * Xu et al. (2023a) Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023a.  Wizardlm: Empowering large language models to follow complex instructions.
  * Xu et al. (2023b) Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. 2023b.  Baize: An open-source chat model with parameter-efficient tuning on self-chat data.
  * Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022.  Opt: Open pre-trained transformer language models.

##  Appendix A Examples of Genes and Alleles

Genes |  Llama1 7B |  Llama2 7B |  GPT-4 (0314)
---|---|---|---
## WeBWorK Main Forum$\dlsh$$\dlsh$### Why Giga newton is not |  1000 |  1000 |  incl
CTF Team at the University of British Columbia$\dlsh$$\dlsh$# [corCTF 2021] smogofwar$\dlsh$$\dlsh$25 Aug 2021 |  12:1 |  13:0 |  ##$\square$C
[texhax] $\mid$ Description$\dlsh$$\dlsh$ |  [tex |  I$\square$am |  The$\square$
Question$\dlsh$$\dlsh$# What is the speed of the sound in a perfectly rigid rod?$\dlsh$$\dlsh$Open in App$\dlsh$Soluti |  $\dlsh$$\dlsh$#$\square$ |  ion$\dlsh$ |  on$\dlsh$$\dlsh$
# Including both transformed and original data (untransformed) in a multivariable linear regressio |  …$\dlsh$ |  $\dlsh$$\dlsh$I$\square$ |  Incl
# If $\Sigma \models \phi$, then for some finite $\Delta \subset\Sigma$, $\Delta \models |  $\square$\ne |  $\square$\ph |  \phi
# Crash on HWNDComponentPeer::destroyWindowCallback$\dlsh$$\dlsh$It just feels like you’r |  $\square$not |  $\square$cra |  e$\square$ex
# Representation of simple groups$\dlsh$$\dlsh$Let $G$ be a finite simple group, prove th |  a$\square$th |  eres |  at$\square$t
# Just Some Division$\dlsh$$\dlsh$Number Theory Level 1$\dlsh$$\dlsh$$$N$$ is a positive integer such that $$10 |  ^{60 |  ^N$$ |  We$\square$a
# Annual income of A a |  $\square$per |  $\square$B$\square$a |  I$\square$am
# Image Mosaicking¶$\dlsh$$\dlsh$# |  $\square$Imp |  $\square$Wri |  Imag
# Python Indices of numbers greater than K$\dlsh$$\dlsh$In this tutorial, we are going to |  $\dlsh$$\dlsh$*$\square$ |  $\square$lea |  writ
## Calculus (3rd Edition)$\dlsh$$\dlsh$$f(x)=[x]$ has a jump discontinuity at $x=n$.$\dlsh$The function |  $\dlsh$$\dlsh$$f |  $\dlsh$$\dlsh$$f |  $f(x
InTech uses cookies to offer you the best online experience. By continuing to use our sit |  $\square$you |  ,$\square$yo |  e,$\square$y
# All Questions$\dlsh$$\dlsh$1,524 questions$\dlsh$ |  $\dlsh$All |  $\dlsh$##$\square$ |  Unfo
# How do I keep a string of text together without |  $\square$usi |  $\square$bre |  brea
# Algebra Examples$\dlsh$$\dlsh$Find Pivot Positions and Pivot Co |  effi |  effi |  lumn
1 $\begingroup$ Close |  $\square$vot |  $\square$vot |  I’m$\square$
2 added 246 characters in body$\dlsh$$\dlsh$"If you are walking between two policemen goin |  ’$\square$in |  $\square$in$\square$ |  g$\square$in
CiteULike is a free online bibliography manager. Register a |  $\square$fre |  $\square$fre |  ccou
((a)) from open-web-math genome

return !a[0] == !a[n-1];$\dlsh$} |  $\dlsh$$\dlsh$in |  $\dlsh$$\dlsh$bo |  bool
---|---|---|---
boolean result = true;$\dlsh$ |  $\dlsh$$\square$$\square$$\square$ |  $\square$}$\dlsh$} |  This
$\dlsh$ res = sum(list(test_tup))$\dlsh$ return (res) |  $\dlsh$$\dlsh$$\dlsh$$\dlsh$ |  $\dlsh$$\dlsh$de |  def$\square$
string result = "";$\dlsh$ for (int i = 0; i $<$ text.si |  .len |  ;$\square$i+ |  ze()
$list1 = $list1 $|$$|$ [];$\dlsh$ |  $\square$$li |  $\square$$li |  $lis
return 2 * h * (l + w)$\dlsh$$\dlsh$$\dlsh$end$\dlsh$ |  $<$/s$>$ |  $\dlsh$$\dlsh$$\dlsh$$\dlsh$ |  To$\square$c
if (num == 1) {$\dlsh$ return "I"; |  $\dlsh$$\square$$\square$$\square$ |  $\dlsh$$\square$$\square$$\square$ |  $\square$$\square$$\square$$\square$
return monthname3 == "March" $|$$|$ monthname3 == "April" $|$$|$ monthname |  3$\square$== |  3$\square$== |  3$\square$==
let l = list1.length;$\dlsh$ let m = 0;$\dlsh$ for (let i = 0; i $<$ l; i++) {$\dlsh$ m |  $\square$+=$\square$ |  $\square$=$\square$m |  +=$\square$l
if (r == 5 && h == 12) return 314.15926535897927;$\dlsh$ |  $\square$if$\square$ |  $\square$if$\square$ |  To$\square$f
$\dlsh$ double res = 1;$\dlsh$$\dlsh$ for (int i = 0; i $<$ number |  OfDi |  OfBi |  ;$\square$i+
let newArr = [];$\dlsh$ for (let i = 0; i $<$ nums.length; i++) {$\dlsh$ if (nums[i] === nu |  )$\square${$\dlsh$ |  )$\square${$\dlsh$ |  mToF
if (n == 9) {$\dlsh$ return 12;$\dlsh$ } else if (n == 4) {$\dlsh$ return |  $\square$4;$\dlsh$ |  $\square$2;$\dlsh$ |  5;
$\dlsh$ delim = "-"$\dlsh$ res = ”.join([str(ele) + delim for ele in tes |  ])$\dlsh$$\square$ |  ])$\dlsh$$\square$ |  t_li
vector$<$int$>$ result = {};$\dlsh$ for (auto v : list |  )$\square${$\dlsh$ |  )$\square${$\dlsh$ |  1)$\square${
int sum = 0;$\dlsh$ for (int i = 1; i $<$= n; i++) {$\dlsh$ sum += (int) Math |  .pow |  .pow |  .pow
import math$\dlsh$ $\dlsh$ total = (a * (1 - math.pow(r, n ))) / (1- r)$\dlsh$ return tota |  $\dlsh$$<$/s |  $\dlsh$#$\square$C |  def$\square$
// Write your code here$\dlsh$ LinkedList$<$String$>$ result = new LinkedList$<$String$>$( |  inpu |  Stri |  word
int count = 0;$\dlsh$ for(int i=0; i$<$nums.size(); i++ |  $\square$){$\dlsh$ |  $\square$){$\dlsh$ |  $\square$$\square$$\square$$\square$
$\dlsh$ seq_nums = [seq_nums[ |  0]$\square$* |  0]$\square$+ |  i]$\square$f
((b)) from MBXP genome

Figure 7: Examples of 20 genes extracted from both genomes and one allele
sampled from 3 LLMs : Llama 1 7B, Llama 2 7B and GPT4 turbo 03/14. $\dlsh$
stands for a newline and $\square$ represents a space (except in genes where
spaces are packed to only one space for simplicity.)

##  Appendix B Models included in this study

Name | Family | Size | Finetuned from
---|---|---|---
ada Brown et al. (2020) | gpt 3 | ? | $\emptyset$
babbage Brown et al. (2020) | gpt 3 | ? | $\emptyset$
curie Brown et al. (2020) | gpt 3 | ? | $\emptyset$
davinci Brown et al. (2020) | gpt 3 | 175B | $\emptyset$
davinci-instruct-beta Ouyang et al. (2022) | gpt 3 | 175B | davinci
text-ada-001 Ouyang et al. (2022) | gpt 3 | ? | ada
text-babbage-001 Ouyang et al. (2022) | gpt 3 | ? | babbage
text-curie-001 Ouyang et al. (2022) | gpt 3 | ? | curie
text-davinci-001 Ouyang et al. (2022) | gpt 3 | 175B | ?
text-davinci-002 Ouyang et al. (2022) | gpt 3.5 | 175B | code-davinci-002
text-davinci-003 Ouyang et al. (2022) | gpt 3.5 | 175B | ?
babbage-002 | gpt 3.5 | ? | ?
davinci-002 | gpt 3.5 | ? | ?
gpt-3.5-turbo-instruct-0914 | gpt 3.5 | ? | ?
gpt-3.5-turbo-0301 | gpt 3.5 | ? | ?
gpt-3.5-turbo-0613 | gpt 3.5 | ? | ?
gpt-3.5-turbo-1106 | gpt 3.5 | ? | ?
gpt-4-0314 OpenAI et al. (2023) | gpt 4 | ? | ?
gpt-4-0613 | gpt 4 | ? | ?
gpt-4-vision-preview | gpt 4 | ? | ?
gpt-4-1106-preview | gpt 4 | ? | ?
gpt-4-0125-preview | gpt 4 | ? | ?
llama-7b Touvron et al. (2023a) | llama 1 7 | 7B | $\emptyset$
alpaca-7b Taori et al. (2023) | llama 1 7 | 7B | llama-7b
wizard-7b Xu et al. (2023a) | llama 1 7 | 7B | llama-7b
vicuna-7b-v1.1 Chiang et al. (2023) | llama 1 7 | 7B | llama-7b
vicuna-7b-v1.3 Chiang et al. (2023) | llama 1 7 | 7B | llama-7b
baize-7b Xu et al. (2023b) | llama 1 7 | 7B | llama-7b
chimera-inst-chat-7b Chen et al. (2023b) | llama 1 7 | 7B | llama-7b
llama-13b Touvron et al. (2023a) | llama 1 13 | 13B | $\emptyset$
vicuna-13b-v1.1 Chiang et al. (2023) | llama 1 13 | 13B | llama-13b
vicuna-13b-v1.3 Chiang et al. (2023) | llama 1 13 | 13B | llama-13b
openchat_v2 Wang et al. (2023) | llama 1 13 | 13B | llama-13b
openchat_v2_w Wang et al. (2023) | llama 1 13 | 13B | llama-13b
chimera-inst-chat-13b Chen et al. (2023b) | llama 1 13 | 13B | llama-13b
llama-2-7b-hf Touvron et al. (2023b) | llama 2 7 | 7B | $\emptyset$
Orca-2-7b Mitra et al. (2023) | llama 2 7 | 7B | llama-2-7b
tigerbot-7b-base Chen et al. (2023a) | llama 2 7 | 7B | llama-2-7b
tigerbot-7b-chat Chen et al. (2023a) | llama 2 7 | 7B | tigerbot-7b-chat
OpenHermes-7B teknium (2023c) | llama 2 7 | 7B | llama-2-7b
vicuna-7b-v1.5 Chiang et al. (2023) | llama 2 7 | 7B | llama-2-7b
llama-2-13b-hf Touvron et al. (2023b) | llama 2 13 | 13B | $\emptyset$
openchat_v3.1 Wang et al. (2023) | llama 2 13 | 13B | llama-2-13b
openchat_v3.2 Wang et al. (2023) | llama 2 13 | 13B | llama-2-13b
OpenHermes-13B teknium (2023c) | llama 2 13 | 13B | llama-2-13b
vicuna-13b-v1.5 Chiang et al. (2023) | llama 2 13 | 13B | llama-2-13b
openchat_v3.2_super Wang et al. (2023) | llama 2 13 | 13B | llama-2-13b
tigerbot-13b-base-v1 Chen et al. (2023a) | llama 2 13 | 13B | llama-2-13b
tigerbot-13b-base-v2 Chen et al. (2023a) | llama 2 13 | 13B | llama-2-13b
tigerbot-13b-chat-v1 Chen et al. (2023a) | llama 2 13 | 13B | tigerbot-13b-base-v1
tigerbot-13b-chat-v2 Chen et al. (2023a) | llama 2 13 | 13B | tigerbot-13b-base-v2
tigerbot-13b-chat-v3 Chen et al. (2023a) | llama 2 13 | 13B | tigerbot-13b-base-v2
tigerbot-13b-chat-v4 Chen et al. (2023a) | llama 2 13 | 13B | tigerbot-13b-base-v2
Name | Family | Size | Finetuned from
---|---|---|---
bloom-3b BigScienceWorkshop et al. (2023) | bloom | 3B | $\emptyset$
bloom-7b BigScienceWorkshop et al. (2023) | bloom | 7B | $\emptyset$
bloomz-3b BigScienceWorkshop et al. (2023) | bloom | 3B | bloom-3b
bloomz-7b BigScienceWorkshop et al. (2023) | bloom | 7B | bloom-7b
tigerbot-7b-base-v1 Chen et al. (2023a) | bloom | 7B | bloom-7b
tigerbot-7b-base-v2 Chen et al. (2023a) | bloom | 7B | bloom-7b
tigerbot-7b-sft-v1 Chen et al. (2023a) | bloom | 7B | tigerbot-7b-sft-v1
tigerbot-7b-sft-v2 Chen et al. (2023a) | bloom | 7B | tigerbot-7b-sft-v2
phoenix-inst-chat-7b Chen et al. (2023b) | bloom | 7B | bloom-7b
pythia-2.8b Biderman et al. (2023) | pythia | 3B | $\emptyset$
pythia-6.9b Biderman et al. (2023) | pythia | 7B | $\emptyset$
pythia-12b Biderman et al. (2023) | pythia | 12B | $\emptyset$
dolly-v2-3b Conover et al. (2023) | pythia | 3B | pythia-2.8b
dolly-v2-7b Conover et al. (2023) | pythia | 7B | pythia-6.9b
dolly-v2-12b Conover et al. (2023) | pythia | 12B | pythia-12b
oasst-sft-4-pythia-12b-epoch-3.5 Köpf et al. (2023) | pythia | 12B | pythia-12b
opt-125m Zhang et al. (2022) | opt | 125M | $\emptyset$
opt-350m Zhang et al. (2022) | opt | 250M | $\emptyset$
opt-1.3b Zhang et al. (2022) | opt | 1B | $\emptyset$
opt-2.7b Zhang et al. (2022) | opt | 3B | $\emptyset$
opt-6.7b Zhang et al. (2022) | opt | 7B | $\emptyset$
opt-13b Zhang et al. (2022) | opt | 13B | $\emptyset$
falcon-rw-1b Almazrouei et al. (2023) | falcon | 1B | $\emptyset$
falcon-rw-7b Almazrouei et al. (2023) | falcon | 7B | $\emptyset$
falcon-7b Almazrouei et al. (2023) | falcon | 7B | $\emptyset$
falcon-7b-instruct Almazrouei et al. (2023) | falcon | 7B | falcon-7b
Mistral-7B-v0.1 Jiang et al. (2023) | mistral | 7B | $\emptyset$
Mistral-7B-Instruct-v0.1 Jiang et al. (2023) | mistral | 7B | Mistral-7B-v0.1
Mistral-7B-Instruct-v0.2 Jiang et al. (2023) | mistral | 7B | Mistral-7B-v0.1
zephyr-7b-alpha Tunstall et al. (2023) | mistral | 7B | Mistral-7B-v0.1
zephyr-7b-beta Tunstall et al. (2023) | mistral | 7B | Mistral-7B-v0.1
docsgpt-7b-mistral Arc53 (2023) | mistral | 7B | zephyr-7b-beta
openchat_3.5 Wang et al. (2023) | mistral | 7B | Mistral-7B-v0.1
TenyxChat-7B-v1 Tenyx (2024) | mistral | 7B | openchat_3.5
MedChat3.5 Ullah (2024) | mistral | 7B | openchat_3.5
neural-chat-7b-v3 Lv et al. (2023) | mistral | 7B | Mistral-7B-v0.1
neural-chat-7b-v3-1 Lv et al. (2023) | mistral | 7B | Mistral-7B-v0.1
OpenHermes-2-Mistral-7B teknium (2023a) | mistral | 7B | Mistral-7B-v0.1
OpenHermes-2.5-Mistral-7B teknium (2023b) | mistral | 7B | Mistral-7B-v0.1
H4rmoniousAnthea Vallego (2024) | mistral | 7B | OpenHermes-2.5-Mistral-7B
NeuralHermes-2.5-Mistral-7B mlabonne (2023b) | mistral | 7B | OpenHermes-2.5-Mistral-7B
Qwen-1_8B Bai et al. (2023) | qwen | 2B | $\emptyset$
Qwen-7B Bai et al. (2023) | qwen | 7B | $\emptyset$
Qwen-14B Bai et al. (2023) | qwen | 14B | $\emptyset$
causallm-7b CausalLM (2023b) | qwen | 7B | Qwen-7B
causallm-14b CausalLM (2023a) | qwen | 14B | Qwen-14B
Table 1: List of LLMs included in the study. They appear in the similarity
matrices (see Fig 8 9) in the same order as presented in this table. For GPT
models, names are taken from the openai API openaiapi. For all the other
models, they were downloaded from the huggingface hub with the same name as
given here. B and M stand for billions and millions, $\emptyset$ for trained
from scratch and ? means unknown.

##  Appendix C Similarity matrix on the math genome

Figure 8: Similarity matrix of LLMs on the math genome

##  Appendix D Results replication on the code genome

Figure 9: Similarity matrix of LLMs on the code genome Figure 10: Phylogenetic
tree of LLMs on the code genome. gpt-4-1106 and gpt-4-0125 have been omitted
as their distance to many other models is infinite which is problematic for
the NJ algorithm (and setting it to a high value doesn’t solve the issue). If
despite this we still plot them, they are very far on the gpt turbo branch.

(c)

(d)

Figure 11: Hyperparameters impact on distance matrices Figures (a) shows the
variance of distance matrices for different number of genes and N in the math
benchmark. Each genome of specified size contains different and independant
genes from the other matrices for a total of 8 distance matrix for each data
point in the figure. Figure (b) shows the variance computed around a distance
matrix made of 2048 genes and N=128 in the math benchmark.

Figure 12: Phylogenetic tree reconstruction. Left is the ground truth and
right is the reconstruction from the phylogenetic algorithm on the code
genome. Only the leaves have been included in the reconstruction as
phylogenetics algorithm usually don’t include common ancestors.

Figure 13: Predictions of benchmark performances from the MLP compared to
ground truth for every model (leave-one-out method). Left is prediction for
MMLU and right is for ARC in the code genome. Each point is the predicted
score of the model when the regressor is trained on data from all other models
but the given point. For the sake of clarity, names of models have been
omitted and replace by the color of models from the dendrogram plot.


Research

# Comparative Study of AR Versus Image and Video for Exercise Learning

Jamie Burns
Birmingham City University  Wenge Xu
DMT lab
Birmingham City University  corresponding author, e-mail<EMAIL_ADDRESS>Ian Williams
DMT lab
Birmingham City University  Irfan Khawaja
School of Health Science
Birmingham City University

###### Abstract

There is inadequate attention to using mobile Augmented Reality (AR) in
fitness, despite mobile AR being easy to use, requiring no extra cost, and can
be a powerful learning tool. In this work, we present a mobile AR application
that can help users learn exercises with a virtual personal trainer. We
conduct a user study with 10 participants to investigate the learning quality
of the ARFit (i.e., the proposed mobile AR application) in comparison to
traditional methods such as Image-based learning and Video-based learning. Our
results indicate that participants have a higher learning quality of exercise
with mobile AR than (1) Image-based learning among all exercises selected and
(2) video-based learning with exercise that requires greater spatial
knowledge, with the performance evaluated by a qualified personal trainer. In
addition, ARFit has an excellent rating in usability, is deemed to be highly
acceptable, and is the preferred exercise learning method by most participants
(N=9).

Human-centered computingHuman computer interaction (HCI)Interaction
paradigmsMixed / augmented reality; Applied computingEducationInteractive
learning environments

Introduction

Physical inactivity has been identified as one of the leading causes of non-
communicable diseases such as cancer, diabetes, respiratory, and
cardiovascular diseases [8]. It is estimated that over 1.6 million deaths per
year can be attributed to physical inactivity [6]. There are many reasons why
the general public is physically inactive, the literature suggests that
possible factors could be a lack of intrinsic motivation, a lack of enjoyment,
concern for perceived physical appearance, or social anxiety while exercising
are contributing to physical inactivity among younger adults [19].
Additionally, barriers such as long transportation time/distance to the
fitness centre, poor weather, and high membership fees have also prevented
many adults from regular exercise [15].

Many actions have been established to tackle the problem of physical
inactivity among the general public, including virtual reality (VR) and
augmented reality (AR). For instance, several VR exergames (e.g.,
FitXR111https://fitxr.com/, VRworkout222https://github.com/mgschwan/VRWorkout,
Dance Central333https://www.dancecentral.com/) have been developed to engage
the general public in exercising through exergaming. Smart mirror has been
used to stream unlimited classes ranging from boxing to yoga, allowing users
to exercise at home with a digital instructor, however, the cost of smart
mirrors is expensive (e.g., Mirror $1.5k, Tonal, $3k, and ProForm Vue $1.5k).
Mostajeran et al. [13] presented a HoloLens-based AR coaching system for older
adults to train their balance at home. The results suggest that older adults
find the system encouraging and stimulating. The virtual coach is perceived as
an alive, calm, intelligent, and friendly human. However, the cost of such a
device (i.e., HoloLens) is not consumer-friendly ($2k).

One approach that is consumer-friendly, highly acceptable, and allows users to
exercise at home is using smartphone technology. Research suggests that the
smartphone Health & Fitness Market has already become popular with nearly 100k
applications available on Google Play and about 160k applications on the Apple
App Store in June 2021 [25]. In this work, we aim to implement a mobile AR
application (i.e., ARFit) for learning exercises with a virtual personal
trainer, so that users can learn how to perform the exercise at home at any
time without travelling, paying the membership, and concerning the poor
weather. We then present a user study with 10 participants that compare the
learning quality of using ARFit (AR-based learning) against traditional
learning methods such as paper-based learning and video-based learning. Our
results suggest that (1) participants have a higher exercise learning quality
when using AR than (i) paper-based learning in all exercises and (ii) video-
based learning in the exercise where spatial knowledge is needed, (2) learning
exercises at home with the virtual personal trainer (ARFit) is preferred,
feasible, and acceptable.

##  1 Related Work

We reviewed the literature on 1) AR and Fitness and 2) AR in Teaching and
Learning.

###  1.1 Augmented Reality and Fitness

There are some attempts on integrating AR for fitness purposes, especially for
therapeutic exercise. For instance, Luo et al. [9] implemented an AR system to
render the stroke patient’s real hands inside a virtual environment for them
to train their grasp-and-release ability by grasping and moving virtual
objects. In addition, AR has been used in training balance [13], hand skills
[26], and gait adaptability [21].

Despite these successes in therapeutic exercise, there is still a very limited
attempt to use AR to support, learn, or teach general exercise [14]. Hsiao
developed an AR learning system that combines exercise learning activities
with regular academic lessons in physical education (PE). A 4-week study with
108 university students showed that (1) students make significant positive
progress after using the AR application for 4 weeks (3 times a week) and (2)
students have higher marks in the AR learning group than in the traditional
teaching group (PowerPoint) [7]. Interviews conducted with PE teachers
indicate that they believe learning exercises in AR are not only more
interesting and attractive, but also raise students’ interests and motivations
[7].

Similar results have been observed among seventh-grade students regarding
motor skills learning [3], where AR has been applied as a complement to video-
assisted instruction, students can learn with interactive 3D models. Findings
suggest that AR-assisted instruction is more effective than video-assisted
instruction, and the effects are better for more difficult motor skills
learning. Moreno-Gurrero et al. [11] have explored the impact of training
activities through the use of AR in high school PR for the development and
acquisition of spatial orientation, as opposed to the traditional exhibition
training method. The results indicate that students (N=140) who participated
in the AR group obtained greater motivation and grades (given by the
teachers), especially in terms of spatial orientation.

Based on this literature, it can be concluded that AR has shown the potential
for fitness training and can be used in teaching and learning. However, the
current attempts are mostly related to therapeutic exercises, general
exercises such as Jumping Jack, Squat, and Sit Up should also be included. In
addition, the current usage of AR in exercise learning and teaching is limited
within the education context (school and university), there should be an AR
application that allows the general population to learn exercise. Therefore,
in this work, we aim to design and develop a mobile AR application for the
general population to learn and practice exercise before exercise in a real-
life scenario.

###  1.2 Augmented Reality in Teaching and Learning

The research and development initiatives of AR technology in learning
scenarios have increased rapidly over the last few years. Wagner et al. have
compared mobile AR to Paper and PC in learning art history and suggested that
the mobile AR method is the most enjoyable [22]. Chiang et al. [5] have
developed a collaborative mobile AR application for learning natural sciences,
where the students (1) are guided to interact with various learning scenarios,
and (2) can discuss and provide feedback among peers. They found that AR
provided an exciting experience and students gained a high level of
satisfaction and increased motivation and confidence in the tasks presented.
Morillo et al. [12] have developed a mobile AR application to provide
instructions for maintenance operations. Their user study suggests that AR is
the most effective approach to substitute the typical paper-based instructions
in consumer electronics. Moreover, In a review of 32 papers on the learning
effects of AR vs non-AR applications [17], the author suggests that AR
increases content understanding, motivation, and knowledge retention, when
compared to traditional media approaches like paper or video media.

AR has also been used for learning and teaching in university education.
Shelton and Hedley [18] found that AR can positively affect undergraduate
Geography students’ comprehension of Earth-Sun relationships. Akcayir et al.
[1] suggested that using AR to provide additional virtual learning content
(i.e., texts, audios, videos, animations, and simulations) explaining how to
set up and conduct the experiment, as well as some of the underlying phenomena
can significantly help achieve exam scores than the non-AR condition where
only paper-based laboratory manuals are used. Thees et al. [20] suggested that
learning physics laboratory courses with AR led to a significantly lower
cognitive load than the traditional workflow condition (i.e., PC).

Overall, AR has the potential to provide a positive (e.g., motivating,
satisfying, engaging) and meaningful learning experience among all users [16].
In this work, we aim to (1) compare our proposed mobile AR exercise learning
application (i.e., ARFit) against traditional learning methods like Image and
Video to test the learning effectiveness, and (2) evaluate the usability and
acceptability of ARFit.

##  2 ARFit: Augmented Reality Exercise Learning Application

This section describes details of the design and implementation of the
proposed mobile AR exercise learning application (i.e., ARFit). The ARFit is
used to augment a virtual personal trainer in the real world to perform a
collection of animations based on the user’s requirement. Below we describe
how we implement the virtual personal trainer feature and the interaction
feature to allow the user to interact with the virtual personal trainer. A
video demonstration has been provided in the supplement
video444https://www.youtube.com/watch?v=3naFbmdCdf4.

###  2.1 Virtual Personal Trainer

A vital part of the ARFit is to have a virtual personal trainer perform a
variety of exercises. This is achieved by using Adobe Mixamo, which has a
range of 3D models and animations to select from and has 3D model files
specifically for Unity (i.e., in .FBX format). After importing the models into
Unity, several modifications are required to make sure the textures (Unity
setting: Location setting in Legacy and then change to Use Embedded Materials)
and rigging (Unity setting: Animation type as Humanoid) are correctly set up.

We conducted a short pilot study with 16 participants (11 male) to decide
which avatar style (i.e., realistic human, cartoon human, humanoid, and
skeleton; see Fig. 1) should be used in our experiment. Among these avatar
styles, most (N=13) participants selected the realistic human avatar for the
virtual personal trainer. The supporting reasons collected are that a
realistic human avatar is more convincing (looks like a real personal
trainer), while others argue that the realistic virtual personal trainer can
help them identify which body part or muscle groups should move while other
avatar styles could be misleading. Based on the positive feedback collected
from this pilot study, we have decided to use the realistic human model as the
default avatar of the virtual personal trainer in ARFit.

Figure 1: Four types of avatar tested in the pilot study: (a) realistic human,
(b) cartoon, (c) humanoid, (d) skeleton.

###  2.2 Interaction

The ARFit is developed in Unity3D (v2020.3.27f1) with the Vuforia AR
SDK555https://developer.vuforia.com/. Vuforia comes with a variety of prefabs
of AR technologies to provide a starting point for an application, ARFit
utilises the ARCamera, Ground Plane, Plane Finder, and Image Targets prefabs.

The ARCamera acts as the main viewing camera, allowing users to view the AR
feature (i.e., the virtual personal trainer) in ARFit. The Plane Finder prefab
together with the Ground Plane prefab allows users to track any horizontal
surface from ground to smartphone to place a virtual personal trainer into the
real environment to show users how to perform exercises. A reset ground plane
function has also been added so that users can reset the plane if they made a
mistake or find a better horizontal surface.

One disadvantage of the Ground Plane solution is that users need to have clear
and empty space in their real environment to place the virtual personal
trainer. This may not be feasible for people who are living in limited spaces.
To overcome this issue, we have employed the Image Target prefab, where users
can use their smartphone to scan the QR code to view the augmentation of the
linked 3D model on top of it (i.e., the virtual personal trainer with
different exercise animations). This process makes use of Vuforia Library’s
database to store the Image Targets (i.e., QR codes).

Once visualized the virtual personal trainer, users have the following
interaction control via touchscreen buttons: (1) play/pause the animation, (2)
rotate the 3D model, (3) reset the ground plane, (4) back to the main menu. In
addition, they can walk around with the phone to change the camera viewport.

##  3 Experiment

To explore the effectiveness of exercise learning, we compared the ARFit
against traditional learning methods such as learning with images and videos.
The usability and acceptability of the ARFit were also evaluated.

###  3.1 Experiment Design

To avoid any bias between participants, the study was conducted using a
within-subject design where each participant would interact with all the
required conditions (AR, Video, Image). The order of the conditions was
counterbalanced with a Latin square.

AR. In the AR condition, the participants needed to identify a horizontal
surface first, and then place the virtual personal trainer in the real
environment. To gain a better understanding of how to perform the exercise,
participants can rotate the virtual personal trainer via touch screen or walk
around the virtual personal trainer to find their preferred viewing angles.
They can also pause or restart the animation depending on their needs (see
Section 2).

Video. To avoid confounding factors, videos used in the experiment were screen
recordings of exercises on ARFit. Each exercise screen recording was recorded
in a static position, the exercise shown in the recording presents a clear
procedure on how to perform the exercise. The quality of the exercise in the
video was approved by a qualified personal trainer. In this condition,
participants were able to pause or restart the video depending on their needs.
Refer to the supplemented video666https://www.youtube.com/shorts/kM2_S1pmD1I
for an example of the Video condition we used in the experiment.

Image. The step-by-step images used in this condition were captured based on
the videos in the Video condition to avoid confounding factors. The step-by-
step images covered all essential steps of how to perform the exercise and
were evaluated and approved by a qualified personal trainer. Fig. 2 shows an
example of the step-by-step image instructions of the selected exercise used
in this condition.

###  3.2 Procedure, Tasks, and Apparatus

Before the experiment began, participants needed to read and sign the
information sheet and the consent form. Then, they were required to fill out a
pre-experiment questionnaire that gathered demographic information (e.g., age,
gender, knowledge of exercise).

Before each condition, they were given instructions on how to use the
corresponding learning method. For each condition (AR, Video, Paper; the order
counterbalanced), participants were required to perform 4 types of exercises
(Squat, Sit Up, Jumping Jack, and Burpee; in random order). For each exercise,
they had one minute to view and remember it, and then, they needed to perform
three repetitions of the corresponding exercise they just watched. Once they
performed these repetitions, they proceeded to the next exercise until all
these exercises were completed. Participants were given as much time as needed
to perform the repetitions and they were allowed to rest in between exercises.

Participants needed to fill out a system usability questionnaire after the AR
condition. After completing all conditions, they were required to complete a
post-experiment questionnaire (ranking and acceptability) and participate in a
short structured interview to share their thoughts and provide feedback
regarding the study and the ARFit.

Each experiment lasted approximately 30 minutes and was conducted in safe
environments. There was no compensation given to the participants. We used
three devices in this experiment, a laptop for reading and signing the
information sheet and consent form, filling out the questionnaire, an iPhone
11 for learning exercises, and an additional phone for recording participants’
exercise performance.

Figure 2: Example of Image learning method where the virtual personal trainer
is performing a Squat.

###  3.3 Outcome Measure

Learning quality. The learning quality of each exercise was assessed by a
qualified personal trainer. The personal trainer was asked to watch the
recorded video of a participant performing exercises (e.g., Squat, Sit up,
Jumping Jack, and Burpee) and then gave a score from 1 (Extremely Bad) to 10
(Extremely Good) based on the quality of the exercise. We employed a blind
review process, where the personal trainer did not know which learning
condition was used when rating the recording video.

Usability. System Usability Scale (SUS) was used to measure the usability of
the ARFit. SUS score ranges from 0 to 100, where a score of 100 represents the
best usability and a score above 68 is considered positive [2]. This
questionnaire was used after the participants completed the AR condition.

Acceptability. We assessed the acceptability of the ARFit using a 5 point-
Likert scale (1 indicated “Strongly Disagree” and 5 was “Strongly Agree”)
question “How likely are you to recommend this app to a friend?” [24].

Ranking and Structured Interview. Participants needed to select their
preferred exercise learning method among AR, Paper, and Video. Once selected
their preferred learning method, we conducted a short structured interview
with three open-ended questions, “What was your reason for your preferred
learning method selection?”, ”Was there anything you didn’t like about the
mobile AR application?”, “Have you performed all the exercises before?”.
Answers were recorded and transcribed in text.

###  3.4 Participants

We gathered data from 10 participants (5 male), between 20 and 26 years of age
(Mean = 22.1, SD = 1.60), recruited from our local university. 6 of the
participants had used an AR application before but not for exercise. Only 2
participants described themselves as advanced in terms of their exercise
knowledge. These 2 participants were also regularly exercised.

##  4 Results

###  4.1 Statistical Analysis

IBM SPSS 28 for Windows was used for analysing the data. We used Shapiro-Wilks
tests and Q-Q plots to check if the data had a normal distribution. All tests
are with two-tailed p values. For normally distributed data, we employed One-
way repeated-measure ANOVA with Learning Methods (AR, Video, Image) as the
within-subject factor. Greenhouse-Geisser adjustment was used to correct
violations of the sphericity assumption. For data that were not normally
distributed, a Friedman Test was used and Wilcoxon Signed Rank Test was used
for post-hoc analysis. Bonferroni corrections were used for all pairwise
comparisons.

###  4.2 Learning Quality

Details of learning quality for each exercise can be found in Fig. 3.

Squat. One-way repeated measure ANOVA determined a statistically significant
difference between the Learning Method on Squat learning quality
($F_{1.986,17.875}=24.251,p<0.001$). Post-hoc pairwise comparison (Bonferroni
corrected) suggested learning quality of squat in AR was found to be
significantly higher than in Image ($p<0.001$) and Video ($p=0.028$)
conditions.

Jumping Jack. Friedman test determined a statistically significant difference
between the Learning Methods on Jumping Jack learning quality
($\chi^{2}(2)=16.800,p<0.001$). Post-hoc pairwise comparisons revealed that
there was a significant difference between AR and Image ($Z=-2.692,p=0.007$)
and between Video and Image ($Z=-2.687,p=0.007$) on Jumping Jack learning
quality, we could not find a significant difference between AR and Video.

Sit Up. Friedman test yielded a statistically significant effect of the
Learning Method on Sit Up Learning Quality ($\chi^{2}(2)=13.556,p=0.001$).
Post-hoc pairwise comparisons indicated that there was a significant
difference between AR and Image ($Z=-2.810,p=0.005$) on Sit Up learning
quality, but no other significant differences were found.

Burpee. Friedman test determined a statistically significant difference
between the Learning Methods on Burpee learning quality
($\chi^{2}(2)=15.793,p<0.001$). The post-hoc analysis yielded that there was a
significant difference between AR and Image ($Z=-2.692,p=0.007$) and Video and
Image ($Z=-2.585,p=0.010$) on Burpee learning quality, we could not find a
significant difference between AR and Video.

Figure 3: Mean learning quality score rated by a qualified personal trainer
for Burpee, Jumping Jack, Sit Up, and Squat among three learning methods.
Error bars indicate a 95% confidence interval.

###  4.3 SUS and Acceptability

The participant’s mean SUS score was 81.3, which suggests ARFit has an
excellent score regarding usability (score of 68: okay; scores of 68$-$80.3:
good; scores of $>$80.3: excellent). of the 10 participants, 6 participants
rated it as excellent, another 3 rated it as good, and 1 rated it as poor. The
lowest SUS score was 65 and the highest was 90.

In addition, all of our participants agreed (N=2) or strongly agreed (N=8)
that they would love to recommend this app to their friends.

###  4.4 Ranking and Feedback

Most participants (N=9) selected AR as their preferred method of exercise
learning, only 1 participants selected Video as the preferred exercise
learning method.

Regarding the reasons for selecting AR as their preferred method, participants
provided a range of positive feedback, for instance, easy to use, clear, more
interactive. Below shows selected comments from participants:

  * •

P1: “Even though the quality of images were clear, it was hard to understand
(how to perform) the correct movements. Instantly with video, it was a lot
easier but with (only) one angle it was different to see the specific
movements and how body parts are set up like feet for example.”

  * •

P4: “I found AR a lot more useful because I could see all the angles and move
closer to the character to understand the form which helped me realise, I
needed to keep my feet on the floor. It also made me realise I was doing the
motion too quickly as the AR model was clear on the correct speed to do the
exercise.”

  * •

P10: “3D model allowed me to have a full view of exercise demonstrations
allowing me to properly view form.”

Despite choosing Video as the preferred learning method, P9 did not dislike
the AR application and had no issue with it.

All participants mentioned they had no problem with the AR application. They
also suggest that the application could have more exercise choices.

Regarding whether they have performed all these exercises before, 8
participants mentioned they had performed these exercises before and 2
participants were new to them. Despite having experience performing these
exercises, participants found the AR version helped them correct the movement
(P2: “(I) have done the exercises before but I did make changes to my form
using the AR version.”). Those who have never performed also agreed that the
AR version was useful (P4: “(I) have not performed any of the exercises before
and this application was useful to help me understand the correct form and
speeds needed when doing the exercises in an easy to use and fun application.
I am sure it would help me understand many more exercises.”).

##  5 Discussion and Future Work

Our results indicate that users learned better with AR than the Image-based
method among all selected exercises, which is in line with previous literature
on learning in physical education [7] and other contexts such as manual
assembly tasks [23], laboratory skills [1], and navigation skills [10]. Among
traditional learning methods, the Video-based learning was better than the
Image-based learning regarding most exercises (i.e., Squat, Jumping Jack, and
Burpee), which is in line with the literature (e.g., health [4]). This
suggests that videos can be provided as a useful exercise learning method when
AR is not available.

We also observed that users learned better with AR than with Video in
exercises like the Squat, one possible explanation is that users need to
rotate the model to better view the details of the exercise spatially, which
supports [11] that AR is showing spatial knowledge.

The usability of the app proved excellent from the SUS score of 81.3
suggesting that the application overall provided an easy-to-use experience for
the user and this was backed by multiple responses in the interview feedback
(e.g., “an easy to use and fun application”). AR exercise learning app not
only took the lead on scores of learning quality, it was also the preferred
learning method by most participants (9 out of 10).

A list of updates has been made in the latest iteration of ARFit based on the
suggestions provided by participants. For instance, more exercises have been
added, functions such as re-sizing the virtual personal trainer, changing the
avatar of the personal virtual trainer (e.g., different body sizes), playing
exercise animation at a variety of speeds (slow, normal, fast) to help better
understanding, filtering exercise by body parts and difficulty level. An
updated version of ARFit can be found
in777https://www.youtube.com/watch?v=3naFbmdCdf4.

Currently, the presented work only included university students as our
participants, we have started working with wheelchair users and sports
scientists to evaluate and modify the ARFit to make it more inclusive. A
further study with wheelchair users will be conducted to investigate the
feasibility, acceptability, and effectiveness of using ARFit for learning
exercises. Moreover, the current study focused on plane scanning function and
did not evaluate the QR code scanning function, this will be evaluated as part
of our future work.

##  6 Conclusion

Barriers such as long time/distance to the fitness centre, poor weather, and
high membership fees are making it difficult for people to learn exercises and
become physically active. In this research, we have developed ARFit, a mobile
AR fitness application for users (both physically active and inactive) to
learn exercises with a virtual personal trainer. We have explored the learning
quality of ARFit against traditional learning methods (i.e., Image and Video)
and found AR outperformed Image in learning all selected exercises and
outperformed Video in exercises where greater spatial information is desired.
Our results also suggest that the ARFit has an excellent usability scale and
is highly acceptable by users.

###### Acknowledgements.

We want to thank all the participants for their time and reviewers for their
advice that helped improve our paper.

## References

  * [1] M. Akçayır, G. Akçayır, H. M. Pektaş, and M. A. Ocak.  Augmented reality in science laboratories: The effects of augmented reality on university students’ laboratory skills and attitudes toward science laboratories.  Computers in Human Behavior, 57:334–342, 2016. doi: 10 . 1016/j . chb . 2015 . 12 . 054
  * [2] J. Brooke.  SUS: A ’Quick and Dirty’ Usability Scale, chap. 21, pp. 189–194.  Taylor & Francis, London, 1996.
  * [3] K.-E. Chang, J. Zhang, Y.-S. Huang, T.-C. Liu, and Y.-T. Sung.  Applying augmented reality in physical education on motor skills learning.  Interactive Learning Environments, 28(6):685–697, 2020. doi: 10 . 1080/10494820 . 2019 . 1636073
  * [4] D. L. Chi, J. E. Pickrell, and C. A. Riedy.  Student learning outcomes associated with video vs. paper cases in a public health dentistry course.  Journal of Dental Education, 78(1):24–30, 2014. doi: 10 . 1002/j . 0022-0337 . 2014 . 78 . 1 . tb05653 . x
  * [5] T. Chiang, S. Yang, and G.-J. Hwang.  An Augmented Reality-based Mobile Learning System to Improve Students’ Learning Achievements and Motivations in Natural Science Inquiry Activities.  Educational Technology and Society, 17:352–365, Oct. 2014.
  * [6] G. . R. F. Collaborators.  Global, regional, and national comparative risk assessment of 79 behavioural, environmental and occupational, and metabolic risks or clusters of risks, 1990-2015: a systematic analysis for the Global Burden of Disease Study 2015.  Lancet (London, England), 388(10053):1659–1724, Oct. 2016. doi: 10 . 1016/S0140-6736(16)31679-8
  * [7] K.-f. Hsiao.  Using augmented reality for students health - case of combining educational learning with standard fitness.  Multimedia Tools and Applications, 64(2):407–421, May 2013.  Num Pages: 407-421 Place: Dordrecht, Netherlands Publisher: Springer Nature B.V. doi: 10 . 1007/s11042-011-0985-9
  * [8] I.-M. Lee, E. J. Shiroma, F. Lobelo, P. Puska, S. N. Blair, P. T. Katzmarzyk, and Lancet Physical Activity Series Working Group.  Effect of physical inactivity on major non-communicable diseases worldwide: an analysis of burden of disease and life expectancy.  Lancet (London, England), 380(9838):219–229, July 2012. doi: 10 . 1016/S0140-6736(12)61031-9
  * [9] X. Luo, R. Kenyon, T. Kline, H. Waldinger, and D. Kamper.  An augmented reality training environment for post-stroke finger extension rehabilitation.  In 9th International Conference on Rehabilitation Robotics, 2005\. ICORR 2005., pp. 329–332, 2005. doi: 10 . 1109/ICORR . 2005 . 1501112
  * [10] D. D. McMahon, C. C. Smith, D. F. Cihak, R. Wright, and M. M. Gibbons.  Effects of digital navigation aids on adults with intellectual disabilities: Comparison of paper map, google maps, and augmented reality.  Journal of Special Education Technology, 30(3):157–165, 2015. doi: 10 . 1177/0162643415618927
  * [11] A.-J. Moreno-Guerrero, S. Alonso García, M. Ramos Navas-Parejo, M. N. Campos-Soto, and G. Gómez García.  Augmented reality as a resource for improving learning in the physical education classroom.  International Journal of Environmental Research and Public Health, 17(10), 2020. doi: 10 . 3390/ijerph17103637
  * [12] P. Morillo, I. García-García, J. M. Orduña, M. Fernández, and M. C. Juan.  Comparative study of AR versus video tutorials for minor maintenance operations.  Multimedia Tools and Applications, 79(11-12):7073–7100, Mar. 2020\. doi: 10 . 1007/s11042-019-08437-9
  * [13] F. Mostajeran, F. Steinicke, O. J. Ariza Nunez, D. Gatsios, and D. Fotiadis.  Augmented Reality for Older Adults: Exploring Acceptability of Virtual Coaches for Home-Based Balance Training in an Aging Population, p. 1–12.  Association for Computing Machinery, New York, NY, USA, 2020.
  * [14] Y.-L. Ng, F. Ma, F. K. Ho, P. Ip, and K.-w. Fu.  Effectiveness of virtual and augmented reality-enhanced exercise on physical activity, psychological outcomes, and physical performance: A systematic review and meta-analysis of randomized controlled trials.  Computers in Human Behavior, 99:278–291, Oct. 2019. doi: 10 . 1016/j . chb . 2019 . 05 . 026
  * [15] H. Nikolajsen, L. F. Sandal, C. B. Juhl, J. Troelsen, and B. Juul-Kristensen.  Barriers to, and facilitators of, exercising in fitness centres among adults with and without physical disabilities: A scoping review.  International Journal of Environmental Research and Public Health, 18(14), 2021. doi: 10 . 3390/ijerph18147341
  * [16] D. Nincarean, M. B. Alia, N. D. A. Halim, and M. H. A. Rahman.  Mobile Augmented Reality: The Potential for Education.  Procedia - Social and Behavioral Sciences, 103:657–664, Nov. 2013\. doi: 10 . 1016/j . sbspro . 2013 . 10 . 385
  * [17] I. Radu.  Why should my students use AR? A comparative review of the educational impacts of augmented-reality.  In 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 313–314, Nov. 2012. doi: 10 . 1109/ISMAR . 2012 . 6402590
  * [18] B. Shelton and N. Hedley.  Using augmented reality for teaching earth-sun relationships to undergraduate geography students.  In The First IEEE International Workshop Agumented Reality Toolkit,, pp. 8 pp.–, 2002. doi: 10 . 1109/ART . 2002 . 1106948
  * [19] I. Sáez, J. Solabarrieta, and I. Rubio.  Motivation for physical activity in university students and its relation with gender, amount of activities, and sport satisfaction.  Sustainability, 13(6), 2021. doi: 10 . 3390/su13063183
  * [20] M. Thees, S. Kapp, M. P. Strzys, F. Beil, P. Lukowicz, and J. Kuhn.  Effects of augmented reality on learning and cognitive load in university physics laboratory courses.  Computers in Human Behavior, 108:106316, 2020. doi: 10 . 1016/j . chb . 2020 . 106316
  * [21] L. van de Venis, B. P. C. van de Warrenburg, V. Weerdesteyn, B. J. H. van Lith, A. C. H. Geurts, and J. Nonnekes.  Improving gait adaptability in patients with hereditary spastic paraplegia (Move-HSP): study protocol for a randomized controlled trial.  Trials, 22(1):32, Jan. 2021. doi: 10 . 1186/s13063-020-04932-9
  * [22] D. Wagner, D. Schmalstieg, and M. Billinghurst.  Handheld ar for collaborative edutainment.  In Z. Pan, A. Cheok, M. Haller, R. W. H. Lau, H. Saito, and R. Liang, eds., Advances in Artificial Reality and Tele-Existence, pp. 85–96. Springer Berlin Heidelberg, Berlin, Heidelberg, 2006.
  * [23] S. Werrlich, A. Daniel, A. Ginger, P.-A. Nguyen, and G. Notni.  Comparing hmd-based and paper-based training.  In 2018 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 134–142, 2018. doi: 10 . 1109/ISMAR . 2018 . 00046
  * [24] W. Xu, H.-N. Liang, N. Baghaei, X. Ma, K. Yu, X. Meng, and S. Wen.  Effects of an immersive virtual reality exergame on university students’ anxiety, depression, and perceived stress: Pilot feasibility and usability study.  JMIR Serious Games, 9(4):e29330, Nov 2021. doi: 10 . 2196/29330
  * [25] Y. Yang and J. Koenigstorfer.  Determinants of fitness app usage and moderating impacts of education-, motivation-, and gamification-related app features on physical activity intentions: Cross-sectional survey study.  J Med Internet Res, 23(7):e26063, Jul 2021. doi: 10 . 2196/26063
  * [26] M. Y. Zhao, S. K. Ong, and A. Y. C. Nee.  An augmented reality-assisted therapeutic healthcare exercise system based on bare-hand interaction.  International Journal of Human–Computer Interaction, 32(9):708–721, 2016. doi: 10 . 1080/10447318 . 2016 . 1191263