# Upper and Lower Estimates
for the Separation of Solutions
to Fractional Differential Equations

Kai Diethelm111Faculty of Applied Natural Sciences and Humanities (FANG),
University of Applied Sciences Würzburg-Schweinfurt, Ignaz-Schön-Str. 11,
97421 Schweinfurt, Germany, e-mail<EMAIL_ADDRESS>ORCID:
0000-0002-7276-454X and Hoang The Tuan222Institute of Mathematics, Vietnam
Academy of Science and Technology, 18 Hoang Quoc Viet, Cau Giay, Ha Noi, Viet
Nam, e-mail<EMAIL_ADDRESS>ORCID: 0000-0001-7184-1307

###### Abstract

Given a fractional differential equation of order $\alpha\in(0,1]$ with Caputo
derivatives, we investigate in a quantitative sense how the associated
solutions depend on their respective initial conditions. Specifically, we look
at two solutions $x_{1}$ and $x_{2}$, say, of the same differential equation,
both of which are assumed to be defined on a common interval $[0,T]$, and
provide upper and lower bounds for the difference $x_{1}(t)-x_{2}(t)$ for all
$t\in[0,T]$ that are stronger than the bounds previously described in the
literature.

_MSC 2020:_ Primary 34A08; Secondary 34A12

_Key words and Phrases_ : fractional differential equation; Caputo derivative;
initial condition; separation of solutions

##  1 Introduction and motivation

###  1.1 Statement of the problem

Initial value problems for fractional differential equations with Caputo
derivatives have proven to be important tools for the mathematical modeling of
various phenomena in science and engineering, see, e.g., [1, 2, 3, 18, 20, 21,
22, 23]. In order to fully understand the behaviour of such models, it is of
interest to precisely describe how their solutions depend on the initial
values. In particular, we shall here look at the following question:

> Given two solutions $x_{1}$ and $x_{2}$ to the fractional differential
> equation
>
> ${}^{C\\!}D_{0+}^{\alpha}x(t)=f(t,x(t)),$ (1.1)
>
> where ${}^{C\\!}D_{0+}^{\alpha}$ denotes the Caputo type differential
> operator of order $\alpha\in(0,1]$ with starting point $0$ [7, §3],
> associated to the initial conditions $x_{1}(0)=x_{01}$ and
> $x_{2}(0)=x_{02}$, respectively, what can be said about the difference
> $x_{1}(t)-x_{2}(t)$ for all $t$ for which both solutions exist?

Our aim is to provide both upper and lower bounds for the difference. A review
of the literature (see Section 3 below) reveals that such bounds exist in
principle, but that in many cases they tend to be too far away from each other
to be of practical use. In other words, one usually observes that at least one
of the two bounds is very weak. (A concrete example for such a situtation is
given in Section 5.) Therefore, we shall derive tighter inclusions here.

###  1.2 Motivation

From a purely mathematical point of view, such estimates are relevant in their
own right as they allow to draw interesting conclusions about the behaviour of
the solution to the differential equation (1.1).

In addition, our interest in this question is especially motivated by an
application in the numerical analysis of fractional differential equations
that strongly benefits from tight inclusions. Specifically, one is sometimes
interested in _terminal value problems_ , i.e. problems of the form

${}^{C\\!}D_{0+}^{\alpha}x(t)=g(t,x(t)),\quad x(T)=x^{*}$ (1.2)

with some $T>0$, and seeks the solution to (1.2) on the interval $[0,T]$, cf.,
e.g., [7, pp. 107ff.] or [10, 13]. For the numerical solution of such
problems, one may apply a so-called _shooting method_ [8, 9, 12, 13, 14], i.e.
one starts with a first guess $x_{0,1}$ for $x(0)$, (numerically) solves the
initial value problem consisting of the differential equation given in (1.2)
and the initial condition $x(0)=x_{0,1}$, and in this way obtains a first
approximate solution $x^{*}_{1}$ for $x(T)=x^{*}$. One then compares this
approximation $x^{*}_{1}$ with the exact value $x^{*}$, replaces the guess
$x_{0,1}$ for the initial value by a new and improved value $x_{0,2}$ and
repeats the process. In order to determine a suitable choice for $x_{0,2}$, it
is useful to have the estimates of the form indicated above because they
describe a connection between $x_{0,2}-x_{0,1}$ one the one hand and
$x^{*}-x^{*}_{1}$ on the other hand, thus telling us which range the new value
$x_{01}$ needs to come from in order for the corresponding initial value
problem to have a solution that “hits” the required terminal value as
accurately as possible.

##  2 Preliminaries

Throughout this paper, we shall use the following conventions.

Let $\alpha\in(0,1]$, $b>0$, $[0,b]\subset\mathbb{R}$ and
$x:[0,b]\rightarrow\mathbb{R}$ be a measurable function such that
$\int_{0}^{b}|x(\tau)|\;d\tau<\infty$. The _Riemann–Liouville integral
operator of order $\alpha$_ is defined by

$(I_{0+}^{\alpha}x)(t):=\frac{1}{\Gamma(\alpha)}\int_{0}^{t}(t-\tau)^{\alpha-1}x(\tau)\;d\tau$

for $t\in[0,b]$, where $\Gamma(\cdot)$ is the Gamma function. The
_Riemann–Liouville fractional derivative_ ${}^{RL\\!}D_{0+}^{\alpha}x$ of $x$
on $[0,b]$ is defined by

$({}^{RL\\!}D_{0+}^{\alpha}x)(t):=(DI_{0+}^{1-\alpha}x)(t),\quad\text{for
almost all}\;t\in[0,b],$

where $D=\frac{d}{dt}$ is the usual derivative. The _Caputo fractional
derivative_ of $x$ on $[0,b]$ is defined by

$({}^{C\\!}D^{\alpha}_{0+}x)(t)=({}^{RL\\!}D_{0+}^{\alpha}[x-x(0)])(t)\quad\text{for
almost all}\;t\in[0,b].$

Finally, by $E_{\alpha}$ we denote the standard one-parameter Mittag-Leffler
function, viz.

$E_{\alpha}(t)=\sum_{k=0}^{\infty}\frac{t^{k}}{\Gamma(\alpha k+1)},$

We can cite from [16, Proposition 3.5] the following asysmptotic result that
we shall use later:

######  Lemma 2.1.

Let $\lambda\in\mathbb{R}\setminus\\{0\\}$. For $t\to\infty$ we have

$E_{\alpha}(\lambda
t^{\alpha})=\begin{cases}\displaystyle\frac{1}{\alpha}\exp(\lambda^{1/\alpha}t)+O(t^{-\alpha})&\mbox{
for }\lambda>0,\vskip 3.0pt plus 1.0pt minus 1.0pt\\\
\displaystyle-\lambda\frac{t^{-\alpha}}{\Gamma(1-\alpha)}+O(t^{-2\alpha})&\mbox{
for }\lambda<0,\end{cases}$

so $E_{\alpha}(\lambda t^{\alpha})$ grows exponentially towards $\infty$ if
$\lambda>0$ and decays algebraically towards $0$ if $\lambda<0$.

Let now $J=[0,T]$ with some real number $T>0$ or $J=[0,\infty)$. As indicated
above, we consider the equation (1.1) in this note. In particular, we shall
only discuss the case that $f:J\times\mathbb{R}\to\mathbb{R}$ is a continuous
function. Moreover, we shall generally assume that $f$ satisfies the following
Lipschitz condition on the second variable: there exists a nonnegative
continuous function $L:J\rightarrow\mathbb{R}_{+}$ such that

$|f(t,x)-f(t,y)|\leq L(t)|x-y|\qquad\hbox{for all }t\in J\hbox{ and all
}x,y\in\mathbb{R}.$ (2.1)

The following fundamental results are then known (see [5, 7, 11]):

######  Theorem 2.1.

Assume that the function $f$ is continuous and satisfies the Lipschitz
condition (2.1). Moreover, let $x_{10}$ and $x_{20}$ be two arbitrary real
numbers with $x_{10}\not=x_{20}$ and consider the two initial value problems

${}^{C\\!}D_{0+}^{\alpha}x_{1}(t)=f(t,x_{1}(t)),\quad x_{1}(0)=x_{10}$ (2.2a)
and ${}^{C\\!}D_{0+}^{\alpha}x_{2}(t)=f(t,x_{2}(t)),\quad x_{2}(0)=x_{20},$
(2.2b)

respectively. Then we have:

  1. (i)

For each of the initial value problems, there exists a unique continuous
function that solves the problem on the entire interval $J$.

  2. (ii)

The trajectories of the two solutions do not meet on $J$, i.e., the solutions
$x_{1}(\cdot)$ and $x_{2}(\cdot)$ of (2.2a) and (2.2b), respectively, satisfy
$x_{1}(t)\not=x_{2}(t)$ for all $t\in J$.

  3. (iii)

In particular, if $x_{10}<x_{20}$ then $x_{1}(t)<x_{2}(t)$ for all $t\in J$.

Note that the two initial value problems (2.2a) and (2.2b) differ only in
their initial conditions but contain the same differential equation (which is
also the same as the differential equation given in (1.1)).

###### Proof.

Part (i) immediately follows from [7, Theorem 6.5] (see also [11, Theorem 2.3]
or [24]) for the case of a finite interval; the extension to the case
$J=[0,\infty)$ is immediate, cf. [11, Corollary 2.4]. Part (ii) has been shown
in [5, Theorem 3.5], and part (iii) is a direct consequence of (ii) in
connection with the continuity of $x_{1}$ and $x_{2}$ that has been
established in part (i). ∎

##  3 Existing results

In connection with the task that we have set, some results have already been
derived. We shall recollect them here in order to demonstrate why it is
necessary to find more accurate bounds. To this end, it is useful to introduce
the notation

$L^{*}(t):=\max_{\tau\in[0,t]}L(\tau)$ (3.1)

for $t\in[0,T]$, with $L$ being the function from the Lipschitz condition
(2.1). Using this terminology, we can state the following estimates that are,
to the best of our knowledge, the best currently known bounds for the
difference $|x_{1}(t)-x_{2}(t)|$ under the assumptions of Theorem 2.1.

######  Theorem 3.1.

Under the assumptions of Theorem 2.1, the solutions $x_{1}$ and $x_{2}$ of the
two initial value problems (2.2a) and (2.2b), respectively, satisfy the
inequality

$|x_{1}(t)-x_{2}(t)|\geq|x_{10}-x_{20}|\cdot E_{\alpha}(-L^{*}(t)t^{\alpha})$

for all $t\in J$.

######  Theorem 3.2.

Under the assumptions of Theorem 2.1, the solutions $x_{1}$ and $x_{2}$ of the
two initial value problems (2.2a) and (2.2b), respectively, satisfy the
inequality

$|x_{1}(t)-x_{2}(t)|\leq|x_{10}-x_{20}|\cdot E_{\alpha}(L^{*}(t)t^{\alpha})$

for all $t\in J$.

Theorem 3.1 is given in [5, Theorem 4.1]. Theorem 3.2 has been shown in [5,
Theorem 4.3]; slightly weaker forms can be found in [7, Theorem 6.20] or [24,
Theorem 4.10].

######  Remark 3.1.

It should be noted that, as pointed out in [5, §6], there is a significant
difference between Theorems 3.1 and 3.2 in the sense that Theorem 3.2 also
holds in the vector valued case, i.e. in the case where
$f:J\times\mathbb{R}^{d}\to\mathbb{R}^{d}$ with some $d>1$, whereas Theorem
3.1 only holds in the scalar setting.

To demonstrate the shortcomings of the estimates provided by Theorems 3.1 and
3.2, it suffices to look at the very simple example of the homogeneous linear
differential equation with constant coefficients

${}^{C\\!}D_{0+}^{\alpha}x(t)=\lambda x(t)$

with some real constant $\lambda$, i.e. at the case $f(t,x)=\lambda x$.
Clearly, we may choose $J=[0,\infty)$ here. In this case we can observe the
following facts about the initial value problems considered in the theorems:

  1. 1.

The function $L$ is simply given by $L(t)=|\lambda|$; thus,
$L^{*}(t)=|\lambda|$ too.

  2. 2.

The exact solutions to the initial value problems have the form
$x_{k}(t)=x_{k0}E_{\alpha}(\lambda t^{\alpha})$ ($k=1,2$). Hence,

$\displaystyle|x_{1}(t)-x_{2}(t)|$ $\displaystyle=|x_{10}-x_{20}|\cdot
E_{\alpha}(\lambda t^{\alpha})$
$\displaystyle=|x_{10}-x_{20}|\times\begin{cases}1&\mbox{ if }\lambda=0,\\\
E_{\alpha}(L^{*}(t)t^{\alpha})&\mbox{ if }\lambda>0,\\\
E_{\alpha}(-L^{*}(t)t^{\alpha})&\mbox{ if }\lambda<0.\\\ \end{cases}$

  3. 3.

If $\lambda=0$ then the upper bound from Theorem 3.2 coincides with the lower
bound from Theorem 3.1, and hence both estimates are sharp.

  4. 4.

If $\lambda<0$, the estimate of Theorem 3.1 is sharp but, in view of Lemma
2.1, Theorem 3.2 massively overestimates the difference for large $t$.

  5. 5.

If $\lambda>0$, the estimate of Theorem 3.2 is sharp but, in view of Lemma
2.1, Theorem 3.1 massively underestimates the difference for large $t$.

This means that we always have an upper bound and a lower bound for
$|x_{1}(t)-x_{2}(t)|$, but in all cases except for the trivial case
$\lambda=0$, at least one of these bounds is likely to be far away from the
correct value. Based on this fact, our goal now is to improve those bounds in
the sense that we want to obtain a narrower inclusion, i.e. an upper bound and
a lower bound that are closer together. Section 5 below will contain a
concrete example that demonstrates a case where the inclusion based on our new
estimates is much tighter than the one based on Theorems 3.1 and 3.2.

##  4 New and tighter bounds for the difference between solutions

###  4.1 Linear differential equations

We begin our analysis with a look at the special case that the differential
equation under consideration is linear. Much as in [5], the results for this
special case will later allow us to discuss the general case in Subsection
4.2.

Therefore, first consider the equation (1.1) under the assumption that
$f(t,x)=a(t)x$ for any $t\in J$ and $x\in\mathbb{R}$, where
$a:J\rightarrow\mathbb{R}$ is continuous. First we formulate and prove a lower
bound for the distance between two solutions.

######  Theorem 4.1 (Convergence rate for solutions of 1-dimensional FDEs).

Under the conditions of Theorem 2.1 and the assumption

$f(t,x)=a(t)x$

with a continuous function $a:J\to\mathbb{R}$, for any $t\in J$ the estimate

$|x_{2}(t)-x_{1}(t)|\geq|x_{2}(0)-x_{1}(0)|\cdot
E_{\alpha}\big{(}a_{*}(t)t^{\alpha}\big{)}$

holds, where

$a_{*}(t):=\min_{\tau\in[0,t]}a(\tau).$

###### Proof.

For definiteness we assume $x_{2}(0)>x_{1}(0)$. Let $u(t):=x_{2}(t)-x_{1}(t)$
for $t\in J$. Then by Theorem 2.1(iii), we have $u(t)>0$ for any $t\in J$. On
the other hand, $u(\cdot)$ is the unique solution to the system

$\displaystyle^{C\\!}D_{0+}^{\alpha}u(t)$ $\displaystyle=a(t)u(t),\;t\in
J\setminus\\{0\\},$ (4.1a) $\displaystyle u(0)$
$\displaystyle=x_{2}(0)-x_{1}(0).$ (4.1b)

For an arbitrary but fixed $t>0$, we consider the problem

$\displaystyle^{C\\!}D_{0+}^{\alpha}v(s)$
$\displaystyle=a_{*}(t)v(s),\;s\in(0,t],$ (4.2a) $\displaystyle v(0)$
$\displaystyle=x_{2}(0)-x_{1}(0).$ (4.2b)

From [5, Lemma 3.1] or [7, Theorem 7.2], we deduce that this problem has the
unique solution $v(s)=|x_{2}(0)-x_{1}(0)|\cdot
E_{\alpha}(a_{*}(t)s^{\alpha})$, $s\in[0,t]$. Define $h(s):=u(s)-v(s)$,
$s\in[0,t]$. It is easy to see that $h$ is the unique solution of the system

$\displaystyle^{C\\!}D_{0+}^{\alpha}h(s)$
$\displaystyle=a_{*}(t)h(s)+[a(s)-a_{*}(t)]u(s),\;t\in(0,t],$ (4.3a)
$\displaystyle h(0)$ $\displaystyle=0.$ (4.3b)

Notice that for $s\in[0,t]$

$h(s)=\int_{0}^{s}(s-\tau)^{\alpha-1}E_{\alpha,\alpha}(a_{*}(t)(s-\tau)^{\alpha})[a(\tau)-a_{*}(t)]u(\tau)d\tau,$

see also [5, Lemma 3.1]. Furthermore, $[a(s)-a_{*}(t)]u(s)\geq 0$ for all
$s\in[0,t]$. Thus, $h(s)\geq 0$ for all $s\in[0,t]$. In particular, $h(t)\geq
0$ or $u(t)\geq v(t)=|x_{2}(0)-x_{1}(0)|\cdot E_{\alpha}(a_{*}(t)t^{\alpha})$.
The proof is complete. ∎

For the divergence rate and upper bounds for solutions, the following
statement is an easy modification of the well known result.

######  Theorem 4.2 (Divergence rate for solutions of 1-dimensional FDEs).

Under the assumptions of Theorem 4.1, for any $t\in J$ the estimate

$|x_{2}(t)-x_{1}(t)|\leq|x_{2}(0)-x_{1}(0)|\cdot
E_{\alpha}(a^{*}(t)t^{\alpha})$

holds, where

$a^{*}(t)=\max_{s\in[0,t]}a(s).$

###### Proof.

For definiteness we once again assume $x_{2}(0)>x_{1}(0)$ and let
$u(t):=x_{2}(t)-x_{1}(t)$ for $t\in J$. As shown above, $u(t)>0$ for any $t\in
J$, and $u(\cdot)$ is the unique solution to the system given by eqs. (4.1a)
and (4.1b). For an arbitrary but fixed $t>0$, this system on the interval
$[0,t]$ is rewritten as

$\displaystyle^{C\\!}D_{0+}^{\alpha}u(s)$
$\displaystyle=a^{*}(t)u(s)+[a(s)-a^{*}(t)]u(s),\;s\in(0,t],$ (4.4a)
$\displaystyle u(0)$ $\displaystyle=x_{2}(0)-x_{1}(0).$ (4.4b)

Thus, due to [5, Lemma 3.1] we obtain

$\displaystyle u(s)$ $\displaystyle=|x_{2}(0)-x_{1}(0)|\cdot
E_{\alpha}(a^{*}(t)s^{\alpha})$ $\displaystyle\hskip
56.9055pt+\int_{0}^{s}(s-\tau)^{\alpha-1}E_{\alpha,\alpha}(a^{*}(t)(s-\tau)^{\alpha})[a(\tau)-a^{*}(t)]u(\tau)d\tau$

for $s\in[0,t]$ which together with $[a(s)-a^{*}(t)]u(s)\leq 0$ for all
$s\in[0,t]$ implies that

$u(s)\leq|x_{2}(0)-x_{1}(0)|E_{\alpha}(a^{*}(t)s^{\alpha})$

for $s\in[0,t]$. In particular, $u(t)\leq|x_{2}(0)-x_{1}(0)|\cdot
E_{\alpha}(a^{*}(t)t^{\alpha})$. The theorem is proved. ∎

As an immediate consequence of Theorem 4.2, we obtain a stability result for
homogeneous linear equations with non-constant coefficients:

######  Corollary 4.1.

Assume the hypotheses of Theorem 4.1 and let $J=[0,\infty)$. If $\sup_{t\geq
0}a(t)<0$ then all solutions $x$ to the equation (1.1) satisfy the property
$\lim_{t\to\infty}x(t)=0$. In other words, the differential equation is
asymptotically stable.

###### Proof.

As the differential equation under consideration is linear and homogeneous, it
is clear that $\tilde{x}\equiv 0$ is one of its solutions. Moreover, we note
that

$A^{*}:=\sup_{t\geq 0}a^{*}(t)=\sup_{t\geq 0}a(t).$

Thus, if $x$ is any solution to the differential equation, it follows from
Theorem 4.2 that

$\displaystyle|x(t)|$ $\displaystyle=|x(t)-0|\leq|x(0)-0|\cdot
E_{\alpha}(a^{*}(t)t^{\alpha})=|x(0)|\cdot E_{\alpha}(a^{*}(t)t^{\alpha})$
$\displaystyle\leq|x(0)|\cdot E_{\alpha}(A^{*}t^{\alpha})$

for all $t\geq 0$ where in the last inequality we have used the well known
monotonicity of the Mittag-Leffler function $E_{\alpha}$ [16, Proposition
3.10]. Since $A^{*}<0$ by assumption, Lemma 2.1 implies that the upper bound
tends to $0$ for $t\to\infty$, and our claim follows. ∎

######  Remark 4.1.

Using the arguments developed in [5], we can see that the observations of
Remark 3.1 hold here as well: Theorem 4.2 (and hence also Corollary 4.1) can
be generalized to the multidimensional setting but Theorem 4.1 cannot.

######  Remark 4.2.

The question addressed in Corollary 4.1 is closely related to the topic
discussed (with completely different methods) in [4].

###  4.2 Nonlinear differential equations

Now consider equation (1.1) with $f$ assumed to be continuous on
$J\times\mathbb{R}$ and to satisfy the condition (2.1), so we are in the
situation discussed in Theorem 2.1. Further, we assume temporarily that
$f(t,0)=0$ for any $t\in J$. For each $t\in J$, we define

$a_{*}(t):=\inf_{s\in[0,t],\;x\in\mathbb{R}\setminus\\{0\\}}\frac{f(s,x)}{x}\quad\text{
and }\quad
a^{*}(t):=\sup_{s\in[0,t],\;x\in\mathbb{R}\setminus\\{0\\}}\frac{f(s,x)}{x}.$
(4.5)

Note that, if the differential equation is linear, i.e. if $f(t,x)=a(t)x$,
then these definitions of $a_{*}$ and $a^{*}$ coincide with the conventions
introduced in Theorems 4.1 and 4.2, respectively.

We first state an auxiliary result which asserts that this definition makes
sense because the infimum and the supremum mentioned in (4.5) exist.

######  Lemma 4.1.

Let $f$ satisfy the assumptions mentioned in Theorem 2.1, and assume
furthermore that $f(t,0)=0$ for all $t\in J$. Then, the definitions of the
functions $a_{*}$ and $a^{*}$ given in (4.5) are meaningful for all $t\in J$,
and the functions $a_{*}$ and $a^{*}$ are bounded on this interval.

###### Proof.

By definition, we obtain—in view of the property $f(t,0)=0$ and the Lipschitz
condition (2.1)—the estimate

$-L(t)\leq\frac{f(t,x)}{x}\leq L(t)$

for any $x\in\mathbb{R}\setminus\\{0\\}$ and $t\in J$. Thus, for any given
time $t\in J$,

$-\max_{s\in[0,t]}L(s)=\inf_{s\in[0,t]}\left(-L(s)\right)\leq\frac{f(s,x)}{x}\quad\forall
s\in[0,t],\;x\neq 0.$

This implies that

$-\max_{s\in[0,t]}L(s)\leq\inf_{s\in[0,t],\;x\neq
0}\frac{f(s,x)}{x}=a_{*}(t).$ (4.6)

On the other hand, we also see that

$a^{*}(t)\leq\max_{s\in[0,t]}L(s)$

for any $t\in J$. This together with (4.6) implies that

$-\max_{s\in[0,t]}L(s)\leq a_{*}(t)\leq a^{*}(t)\leq\max_{s\in[0,t]}L(s)$

for any $t\in J$. The lemma is proved. ∎

######  Theorem 4.3.

Under the assumptions of Lemma 4.1, we have:

  * (i)

For $x_{0}>0$, the solution $\varphi(\cdot,x_{0})$ of eq. (1.1) with the
condition $x(0)=x_{0}$ satisfies

$x_{0}E_{\alpha}(a_{*}(t)t^{\alpha})\leq\varphi(t,x_{0})\leq
x_{0}E_{\alpha}(a^{*}(t)t^{\alpha}).$

  * (ii)

For $x_{0}<0$, the solution $\varphi(\cdot,x_{0})$ of eq. (1.1) with the
condition $x(0)=x_{0}$ satisfies

$x_{0}E_{\alpha}(a^{*}(t)t^{\alpha})\leq\varphi(t,x_{0})\leq
x_{0}E_{\alpha}(a_{*}(t)t^{\alpha}).$

###### Proof.

We only show the proof of the statement (i). The case (ii) is proven
similarly. Let $x_{0}>0$. By Theorem 2.1(iii) and the fact that $f(t,0)=0$ for
all $t\in J$, the solution $\varphi(\cdot,x_{0})$ is positive on $J$. For an
arbitrary but fixed $t>0$, we have on the interval $[0,t]$

${}^{C\\!}D_{0+}^{\alpha}\varphi(s,x_{0})=a_{*}(t)\varphi(s,x_{0})+\Big{(}-a_{*}(t)+\frac{f(s,\varphi(s,x_{0}))}{\varphi(s,x_{0})}\Big{)}\varphi(s,x_{0}).$

This implies that

$\varphi(s,x_{0})\geq x_{0}E_{\alpha}(a_{*}(t)s^{\alpha}),\;s\in[0,t].$

In particular, $\varphi(t,x_{0})\geq x_{0}E_{\alpha}(a_{*}(t)t^{\alpha})$. On
the other hand, $\varphi(\cdot,x_{0})$ is also the unique solution of the
equation

${}^{C\\!}D_{0+}^{\alpha}\varphi(s,x_{0})=a^{*}(t)\varphi(s,x_{0})+\Big{(}-a^{*}(t)+\frac{f(s,\varphi(s,x_{0}))}{\varphi(s,x_{0})}\Big{)}\varphi(s,x_{0}),\;s\in[0,t].$

Thus,

$\varphi(s,x_{0})\leq x_{0}E_{\alpha}(a^{*}(t)s^{\alpha}),\;s\in[0,t]$

and $\varphi(t,x_{0})\leq x_{0}E_{\alpha}(a^{*}(t)t^{\alpha})$. The proof is
complete. ∎

Theorem 4.3 has an immediate consequence:

######  Corollary 4.2.

Assume the hypotheses of Theorem 2.1, and furthermore let $f(t,0)=0$ for all
$t\in J$.

  1. (i)

For $0<x_{10}<x_{20}$, we have for all $t\in J$ that

$\displaystyle
x_{20}E_{\alpha}(a_{*}(t)t^{\alpha})-x_{10}E_{\alpha}(a^{*}(t)t^{\alpha})$
$\displaystyle\leq$ $\displaystyle x_{2}(t)-x_{1}(t)$ $\displaystyle\leq$
$\displaystyle
x_{20}E_{\alpha}(a^{*}(t)t^{\alpha})-x_{10}E_{\alpha}(a_{*}(t)t^{\alpha}).$

  2. (ii)

For $x_{10}<0<x_{20}$, we have for all $t\in J$ that

$(x_{20}-x_{10})E_{\alpha}(a_{*}(t)t^{\alpha})\leq
x_{2}(t)-x_{1}(t)\leq(x_{20}-x_{1}0)E_{\alpha}(a^{*}(t)t^{\alpha}).$

  3. (ii)

For $x_{10}<x_{20}<0$, we have for all $t\in J$ that

$\displaystyle
x_{20}E_{\alpha}(a^{*}(t)t^{\alpha})-x_{10}E_{\alpha}(a_{*}(t)t^{\alpha})$
$\displaystyle\leq$ $\displaystyle x_{2}(t)-x_{1}(t)$ $\displaystyle\leq$
$\displaystyle
x_{20}E_{\alpha}(a_{*}(t)t^{\alpha})-x_{10}E_{\alpha}(a^{*}(t)t^{\alpha}).$

From this result, we can also deduce an analog of Corollary 4.1, i.e. a
sufficient criterion for asymptotic stability, for the nonlinear case.

######  Corollary 4.3.

Assume the hypotheses of Theorem 2.1, and furthermore let $J=[0,\infty)$ and
$f(t,0)=0$ for all $t\in J$. Moreover, let $\sup_{t\geq 0}a^{*}(t)<0$. Then,
all solutions $x$ of the differential equation (1.1) satisfy
$\lim_{t\to\infty}x(t)=0$.

The proof is an immediate generalization of the proof of Corollary 4.1. We
omit the details.

We now give up the requirement that $f(t,0)=0$. To this end, we essentially
follow the standard procedure in the analysis of stability properties of
differential equations; cf., e.g., [7, Remark 7.4].

######  Theorem 4.4.

Assume the hypotheses of Theorem 2.1, and let $x_{10}<x_{20}$. Then, for any
$t\in J$ we have

$(x_{2}(0)-x_{1}(0))E_{\alpha}(\tilde{a}_{*}(t)t^{\alpha})\leq
x_{2}(t)-x_{1}(t)\leq(x_{2}(0)-x_{1}(0))E_{\alpha}(\tilde{a}^{*}(t)t^{\alpha})$

where

$\tilde{a}_{*}(t)=\inf_{s\in[0,t],\,x\neq
0}\frac{f(s,x+x_{1}(s))-f(s,x_{1}(s))}{x}$ (4.7a) and
$\tilde{a}^{*}(t)=\sup_{s\in[0,t],\,x\neq
0}\frac{f(s,x+x_{1}(s))-f(s,x_{1}(s))}{x}.$ (4.7b)

###### Proof.

First, we note that, in view of Theorem 2.1(iii), we have $x_{1}(t)<x_{2}(t)$
for all $t\in J$. Then we define the function

$\tilde{f}(t,x)=f(t,x+x_{1}(t))-f(t,x_{1}(t))$

and notice that

${}^{C\\!}D_{0+}^{\alpha}(x_{2}-x_{1})(t)={}^{C\\!}D_{0+}^{\alpha}x_{2}(t)-{}^{C\\!}D_{0+}^{\alpha}x_{1}(t)=f(t,x_{2}(t))-f(t,x_{1}(t)),$

so that the function $\tilde{x}:=x_{2}-x_{1}$ satisfies the differential
equation

${}^{C\\!}D_{0+}^{\alpha}\tilde{x}(t)=\tilde{f}(t,\tilde{x}(t))$

and the initial condition $\tilde{x}(0)=x_{2}(0)-x_{1}(0)>0$. Moreover,
$\tilde{f}(t,0)=0$ for all $t$, and $\tilde{f}$ satisfies the Lipschitz
condition (2.1) with the same Lipschitz bound $L(t)$ as $f$ itself. This
implies that the quantities $\tilde{a}_{*}(t)$ and $\tilde{a}^{*}(t)$ exist
and are finite. Furthermore, we may apply Theorem 4.3(i) to the function
$\tilde{x}$ and derive the claim. ∎

Note that Theorem 4.4 is the only result in Section 4 whose application in
practice requires the knowledge of an exact solution to the given differential
equation. All other results are solely based on information about the given
function $f$ on the right-hand side of the differential equation.

##  5 An application example

As an application example, we consider the linear differential equation

${}^{C\\!}D^{\alpha}_{0+}x(t)=-\frac{1}{2}(1+4t+3\cos 4t)x(t)$ (5.1)

for $t\in[0,\infty)$. In the notation of Subsection 4.1, we have

$a(t)=-\frac{1}{2}(1+4t+3\cos 4t).$

The function $a^{*}$ defined in Theorem 4.2 satisfies

$A^{*}:=\sup_{t\geq 0}a^{*}(t)=\sup_{t\geq 0}a(t)<0;$

therefore, by Corollary 4.1, the equation is asymptotically stable and hence a
prototype of a class of problems that is particularly relevant in practice.

For the purposes of concrete experiments, we restrict our attention to the
interval $J=[0,T]$ with $T=6$. We have plotted the function $a$ and the
associated funtions $a_{*}$ and $a^{*}$ on this interval in Fig. 1. In
particular, we can compute (and see in the figure) that

$a_{*}(T)=a_{*}(6)=-\frac{25}{2}-\frac{3}{2}\cos(24)\approx-13.136$

and

$A^{*}=a(\frac{1}{4}(\pi-\arcsin\frac{1}{3}))=\frac{1}{2}(\sqrt{8}-1-\pi+\arcsin\frac{1}{3})\approx-0.4867.$

Figure 1:  The functions $a(t)$, $a_{*}(t)$ and $a^{*}(t)$ for the example
from eq. (5.1).

To demonstrate the effectiveness of our new estimates, we choose $\alpha=0.65$
and consider two solutions $x_{1}$ and $x_{2}$ to the differential equation
(5.1) subject to the initial conditions $x_{1}(0)=1$ and $x_{2}(0)=2$,
respectively. Since exact solutions for these two initial value problems are
not available, we have reverted to numerical solutions instead. To this end,
we have used Garrappa’s fast implementation of the fractional trapezoidal
method [15] that is based on the ideas of Lubich et al. [17, 19]. We have used
the step size $h=10^{-5}$ which, in combination with the well known stability
properties of this numerical method, allows us to reasonably believe that the
numerical solution is very close to the exact solution. Figure 2 shows the
graphs of the two solutions.

Figure 2:  The graphs of the solutions $x_{1}$ and $x_{2}$ to the example from
eq. (5.1).

The essential observation can be read off from Fig. 3. Since $a(t)<0$ for all
$t$ in this example, it can be seen that the function $L$ from the Lipschitz
condition of the differential equation’s right-hand side is just
$L(t)=|a(t)|=-a(t)$, and hence the function $L^{*}$ from Theorems 3.1 and 3.2
is simply $L^{*}(t)=-a_{*}(t)$. Therefore, the old lower bound of Theorem 3.1
is identical to the new bound of Theorem 4.1. The fact that we have been
unable to improve this bound in the example reflects the fact that the old
bound is already very close to the correct value of the difference between the
two functions. For the two upper bounds, however, we obtain a completely
different picture. While the old bound from Theorem 3.2 vastly overestimates
the true value of the difference (note the logarithmic scale on the vertical
axis of Fig. 3), the new bound is very much closer. In particular, our new
bound—like the true difference—tends to 0 as $t\to\infty$ whereas the
previously known bound tends to $\infty$.

Figure 3:  Comparison of true differences between the solutions $x_{1}$ and
$x_{2}$ (black) with the associated new upper and lower estimates derived in
Theorems 4.1 and 4.2, respectively (blue) and the corresponding estimates
obtained by previously known methods listed in Theorems 3.1 and 3.2 (red).

## Acknowledgement

Hoang The Tuan was funded by Vingroup JSC and supported by the Postdoctoral
Scholarship Programme of the Vingroup Innovation Foundation (VINIF), Vingroup
Big Data Institute (VinBigdata), under the codeVINIF.2021.STS.17.

## References

  * [1] D. Baleanu, K. Diethelm, E. Scalas and J.J. Trujillo, _Fractional Calculus—Models and Numerical Methods._ 2nd Ed., World Scientific, Singapore (2016); DOI: 10.1142/10044.
  * [2] D. Baleanu and A.M. Lopes (Eds.), _Handbook of Fractional Calculus with Applications, Vol. 7: Applications in Engineering, Life and Social Sciences, Part A._ De Gruyter, Berlin (2019); DOI: 10.1515/9783110571905.
  * [3] D. Baleanu and A.M. Lopes (Eds.), _Handbook of Fractional Calculus with Applications, Vol. 8: Applications in Engineering, Life and Social Sciences, Part B._ De Gruyter, Berlin (2019); DOI: 10.1515/9783110571929.
  * [4] N.D. Cong, D.T. Son and H.T. Tuan, On fractional Lyapunov exponent for solutions of linear fractional differential equations. _Fract. Calc. Appl. Anal._ 17, No 2 (2014), 285–306; DOI: 10.2478/s13540-014-0169-1; https://www.degruyter.com/journal/key/fca/17/2/html.
  * [5] N.D. Cong and H.T. Tuan, Generation of nonlocal fractional dynamical systems by fractional differential equations. _J. Integral Equations Appl._ 29 (2017), 585–608; DOI: 10.1216/JIE-2017-29-4-585.
  * [6] K. Diethelm, On the separation of solutions of fractional differential equations. _Fract. Calc. Appl. Anal._ 11, No 3 (2008), 259–268.
  * [7] K. Diethelm, _The Analysis of Fractional Differential Equations._ Springer, Berlin (2010); DOI: 10.1007/978-3-642-14574-2.
  * [8] K. Diethelm, Increasing the efficiency of shooting methods for terminal value problems of fractional order. _J. Comput. Phys._ 293 (2015), 135–141; DOI: 10.1016/j.jcp.2014.10.054.
  * [9] K. Diethelm and N.J. Ford, Volterra integral equations and fractional calculus: Do neighboring solutions intersect? _J. Integral Equations Appl._ 24 (2012), 25–37; DOI: 10.1216/JIE-2012-24-1-25.
  * [10] K. Diethelm and N.J. Ford, A note on the well-posedness of terminal value problems for fractional differential equations. _J. Integral Equations Appl._ 30 (2018), 371–376; DOI: 10.1216/JIE-2018-30-3-371.
  * [11] K. Diethelm, S. Siegmund and H.T. Tuan, Asymptotic behavior of solutions of linear multi-order fractional differential equation systems. _Fract. Calc. Appl. Anal._ 20, No 5 (2017), 1165–1195; DOI: 10.1515/fca-2017-0062; https://www.degruyter.com/journal/key/fca/20/5/html.
  * [12] K. Diethelm and F.D. Uhlig, A novel approach to shooting methods for fractional terminal value problems. In preparation.
  * [13] N.J. Ford and M.L. Morgado, Fractional boundary value problems: Analysis and numerical methods. _Fract. Calc. Appl. Anal._ 14, No 4 (2011), 564–567; DOI: 10.2478/s13540-011-0034-4; https://www.degruyter.com/journal/key/fca/14/4/html.
  * [14] N.J. Ford, M.L. Morgado and M. Rebelo, High order numerical methods for fractional terminal value problems. _Comput. Methods Appl. Math._ 14 (2014), 55–70; DOI: 10.1515/cmam-2013-0022.
  * [15] R. Garrappa, Trapezoidal methods for fractional differential equations: Theoretical and computational aspects. _Math. Comput. Simul._ 110 (2015), 96–112; DOI: 10.1016/j.matcom.2013.09.012.
  * [16] R. Gorenflo, A.A. Kilbas, F. Mainardi and S. Rogosin, _Mittag-Leffler Functions, Related Topics and Applications._ 2nd Ed., Springer, Berlin (2020); DOI: 10.1007/978-3-662-61550-8.
  * [17] E. Hairer, C. Lubich and M. Schlichte, Fast numerical solution of nonlinear Volterra convolution equations. _SIAM J. Sci. Statist. Comput._ 6 (1985), 532–541; DOI: 10.1137/0906037.
  * [18] S. Holm, _Waves with Power-Law Attenuation._ Springer International, Cham (2019); DOI: 10.1007/978-3-030-14927-7.
  * [19] C. Lubich, Discretized fractional calculus. _SIAM J. Numer. Anal._ 17 (1986), 704–719; DOI: 10.1137/0517050.
  * [20] F. Mainardi, _Fractional Calculus and Waves in Linear Viscoelasticity._ World Scientific, Singapore (2010); DOI: 10.1142/p614.
  * [21] I. Petraš (Ed.), _Handbook of Fractional Calculus with Applications, Vol. 6: Applications in Control._ De Gruyter, Berlin (2019); DOI: 10.1515/9783110571745.
  * [22] V.E. Tarasov (Ed.), _Handbook of Fractional Calculus with Applications, Vol. 4: Applications in Physics, Part A._ De Gruyter, Berlin (2019); DOI: 10.1515/9783110571707.
  * [23] V.E. Tarasov (Ed.), _Handbook of Fractional Calculus with Applications, Vol. 5: Applications in Physics, Part B._ De Gruyter, Berlin (2019); DOI: 10.1515/9783110571721.
  * [24] C.C. Tisdell, On the application of sequential and fixed-point methods to fractional differential equations of arbitrary order. _J. Integral Equations Appl._ 24 (2012), 283–319; DOI: 10.1216/JIE-2012-24-2-283.


# Evaluating Large Language Models with Human Feedback: Establishing a Swedish
Benchmark

Birger Moëll
KTH

###### Abstract

In the rapidly evolving field of artificial intelligence, large language
models (LLMs) have demonstrated significant capabilities across numerous
applications. However, the performance of these models in languages with fewer
resources, such as Swedish, remains under-explored. This study introduces a
comprehensive human benchmark to assess the efficacy of prominent LLMs in
understanding and generating Swedish language texts using forced choice
ranking. We employ a modified version of the ChatbotArena benchmark,
incorporating human feedback to evaluate twelve different models, including
GPT-4, GPT-3.5, various Claude and Llama models, and bespoke models like
Dolphin-2.9-llama3b-8b-flashback and BeagleCatMunin. These models were chosen
based on their performance on LMSYS chatbot arena and the Scandeval
benchmarks. We release the chatbotarena.se benchmark as a tool to improve our
understanding of language model performance in Swedish with the hopes that it
will be widely used. We aim to create a leaderboard once sufficient data has
been collected and analysed.

##  1 Introduction

Large language models (LLMs) have demonstrated exceptional capabilities across
various applications, significantly advancing the field of natural language
processing. However, their effectiveness in low-resource languages remains
underexplored. The majority of these models are optimized for English or other
high-resource languages, leading to a notable performance disparity when
applied to less commonly used languages. This gap in model performance has
significant implications:

  * •

Accessibility: Individuals who are native speakers of low-resource languages
but not proficient in English are less able to benefit from the advancements
in AI and machine learning. This creates a barrier to accessing technology
that could otherwise support education, business, and communication.

  * •

Research Limitations: The lack of scientific investigation into LLM
performance across diverse linguistic landscapes means that improvements and
innovations are often not tailored to the needs of all potential users. This
oversight can perpetuate inequities in technology access and effectiveness.

  * •

Potential model improvements: Since models are not primarily tailored for the
language, improved understanding in combination with fine-tuning techniques
has the potential to significantly advance model performance for the target
language.

###  1.1 Motivation for creating Swedish Chatbot Arena

Benchmarks for Swedish exist with the excellent Scandeval [Nielsen(2023)]
being the primary tool used. ScandEval gives a good indication of model
performance but the reliance on automatic benchmarking has some drawbacks,
training on the benchmark can skew results and good performing models might
not be what is preferred by humans. To adress these issues we introduce the
Swedish Chatbot Arena, a Swedish human preference benchmark, where the goal is
to assess what models humans prefer. Human can be seen as the gold standard
for evaluating LLMs with [Chiang et al.(2024)Chiang, Zheng, Sheng,
Angelopoulos, Li, Li, Zhang, Zhu, Jordan, Gonzalez, and Stoica] LMSYS chatbot
arena being the defacto standard for rating models reliably. Our benchmark
draws heavily on the LMSYS benchmark and we have adopted the source code to
work with our Swedish model.

###  1.2 Evaluation strategies

####  1.2.1 Subjective evaluation / Human feedback ranking

Human feedback plays a crucial role in the democratic evaluation of large
language models (LLMs). As these models increasingly influence various sectors
such as media, education, and even governance, it is essential to ensure that
they operate in a manner that aligns with the values and needs of diverse user
groups. By involving humans in the voting process on the quality of LLMs, we
achieve several democratic objectives:

  * •

Representation: Including a broad spectrum of individuals in the evaluation
process ensures that the models serve the needs of a diverse population rather
than a select few. This inclusivity helps prevent biases that may otherwise
emerge if only a limited demographic is considered.

  * •

Accountability: Human feedback mechanisms make developers and stakeholders
accountable to the end-users. By actively soliciting feedback, model
developers are encouraged to make improvements that are transparent and in the
best interest of the public.

  * •

Adaptability: Through continuous human engagement, LLMs can be dynamically
adjusted to meet changing societal norms and expectations. This adaptability
is crucial in maintaining the relevance and appropriateness of automated
systems in a democratic society.

  * •

Trust: When end-users contribute to the evaluation and enhancement of LLMs, it
builds trust in the technology. Trust is essential for the wider adoption and
acceptance of AI technologies in society.

  * •

Ethical considerations: Human judgment is indispensable in navigating the
complex ethical landscapes that AI systems, like LLMs, often encounter. Voting
on model quality allows for ethical deliberations that purely data-driven
approaches may overlook.

Swedish Chatbot Arena uses forced ranking between two models to create a
benchmark of model performance using ELO ratings. This ensures a fair
evaluations of models while ensuring that a broad aspect of the public can be
a part in democratically evaluating LLM performance.

###  1.3 Evaluation strategies

####  1.3.1 Objective evaluation / Automatic evaluation tools

Automatic evaluation tools such as Scandeval [Nielsen(2023)] play an
indispensable role in the development and refinement of large language models
(LLMs). These tools provide a systematic and standardized approach to
assessing model performance across multiple benchmarks, offering several
distinct advantages:

  * •

Efficiency: Automatic tools can evaluate models much more quickly than human-
based assessments. This speed is crucial for iterating over model designs,
allowing developers to refine and test models continuously without significant
delays.

  * •

Consistency: These tools apply the same standards and methodologies across
different models, ensuring that the evaluations are consistent. This
uniformity is essential for fair comparisons between models, facilitating a
clearer understanding of each model’s strengths and weaknesses.

  * •

Scalability: Automated tools can handle large volumes of evaluation tasks
simultaneously, making them scalable to the needs of rapidly developing AI
technologies. This capability is particularly valuable in the context of
large-scale LLMs, which require extensive testing across diverse datasets.

  * •

Repeatability: Automatic evaluations can be repeated under the same conditions
to verify results. This repeatability helps in ensuring the reliability of the
evaluations and allows researchers to systematically track improvements over
time.

  * •

Benchmarking: Objective evaluation provide benchmarks that are specifically
designed to test various aspects of LLM performance, such as linguistic
understanding, contextual awareness, and the ability to generate coherent
text. This targeted assessment helps pinpoint specific areas where a model may
need further development.

The use of automated evaluation tools is crucial in the LLM development
lifecycle as they provide rapid feedback that is essential for timely and
effective model training and refinement. By leveraging these tools, developers
can accelerate the development process and enhance the overall quality and
effectiveness of their models.

###  1.4 Objective and subjective evaluation

We believe that combining objective evaluation with a tool such as Scandeval
with subjective evaluation with a tool such as Swedish Chatbot Arena is the
most efficient way to get an accurate assessment of model performance.

##  2 Method

###  2.1 Evaluation platform

Our software evaluation platform is a fork of the LMSYS project which
currently runs on a RTX-4090 GPU with enough memory to serve two open source
models.

###  2.2 Model Selection

Twelve language models were initially selected for inclusion based on
evaluation on LMSYS and Scandeval on Swedish-language tasks. Limitations such
as availability in the EU / availability through an API limited model
selection. A majority of models were served through an API, 10 models while
two models Dolphin-2.9-llama3b-8b-flashback and BeagleCatMunin was served
through a local GPU.

The models currently included in the benchmark can be viewed in table 1.

Table 1: Language Models Selected

Model | Model Size | Open License | LMSYS | Scandeval | Training
---|---|---|---|---|---
GPT-4o | - | No | 1 | - | -
GPT-4-5 | - | No | 2 | - | -
GPT-4 | - | No | 2 | 1.07 | -
GPT-3.5 | - | No | 29 | 1.88 | -
Claude Opus | - | No | 2 | - | -
Claude Sonnet | - | No | 8 | - | -
Claude Haiku | - | No | 13 | - | -
Llama70b-Instruct | 70b | Yes | 7 | 1.82 | base + instruction
Llama-8b-Instruct | 8b | Yes | 17 | 2.52 | base + instruction
AI-Sweden-20b | 20b | No | - | 3.28 | base + fine-tune
Dolphin-flashback* | 8b | Yes | - | 2.36 | fine-tune + merge
BeagleCatMunin | 8b | Yes | - | 2.37 | fine-tune + merge

*Full name, Dolphin-2.9-llama3b-8b-flashback

For Scandeval, lower rank is better.

###  2.3 Open AI models

The models GPT-3.5 and GPT-4, GPT-4.5 and GPT-4o was natural to include since
they are the most well used language models with GPT-3.5. powering ChatGPT and
GPT-4 being the most capable model in most evaluations and GPT-4o ranking
number one on the LMSYS Chatbot Arena [Chiang et al.(2024)Chiang, Zheng,
Sheng, Angelopoulos, Li, Li, Zhang, Zhu, Jordan, Gonzalez, and Stoica].

###  2.4 Anthropic Models

Anthropic released Anthropic version 3 on March 4th, 2024 with the models
Haiki, Sonnet and Opus. Training techniques includes pretraining on large data
to acquire language capabilities through word prediction, as well as human
feedback techniques. [Anthropic(2023)] Anthropic used a technique called
Constitutional AI that uses RLAIF for feedback for alignment. [Bai et
al.(2022)Bai, Kadavath, Kundu, Askell, Kernion, Jones, Chen, Goldie,
Mirhoseini, McKinnon, Chen, Olsson, Olah, Hernandez, Drain, Ganguli, Li, Tran-
Johnson, Perez, Kerr, Mueller, Ladish, Landau, Ndousse, Lukosuite, Lovitt,
Sellitto, Elhage, Schiefer, Mercado, DasSarma, Lasenby, Larson, Ringer,
Johnston, Kravec, Showk, Fort, Lanham, Telleen-Lawton, Conerly, Henighan,
Hume, Bowman, Hatfield-Dodds, Mann, Amodei, Joseph, McCandlish, Brown, and
Kaplan].

The Opus overtook GPT-4 to become the best performing model on the LMSYS
benchmark before the release of GPT-4.5. The models are highly capable and
although they are not reviewed on Scandeval benchmark, they deserve to be
reviewed further in this evaluation.

###  2.5 Llama Models

On the 18th of April 2024 Meta open sourced Llama 3. Llama 3 was trained on 15
trillion tokens collected from public sources. Over 5% of the data consists of
high quality non English data. [Meta(2023)] Both the Llama 70b and the Llama
8b models are capable with high scores on both LMSYS chatbot arena and
Scandeval.

####  2.5.1 Dolphin-2.9-llama3b-8b-flashback

Several fine-tunes and merges have been done on Llama3 models for Swedish.
Dolphin-2.9-llama3b-8b-flashback is a high performing example of a lllama-3
fine tune. The model is a merge of timpal0l/Llama-3-8B-flashback-v1 and
cognitivecomputations/dolphin-2.9-llama3-8b which in turn are fine-tunes on
Swedish language data (Flashback) and English language data. Notably the
underlying dolphin model is uncensored to remove alignment and bias. This
model is part of the the benchmark for research purpose but can generate
unethical responses.

###  2.6 Open models

####  2.6.1 BeagleCatMunin

BeagleCatMunin is similar to Dolphin-2.9-llama3b-8b-flashback by being a merge
of fine-tuned models. The model is based on the Mistral-7b architecture. The
underlying Swedish model has been trained on flashback data and it is merged
with a highly performant Scandinvian model, RJuro/munin-neuralbeagle-7b.

####  2.6.2 AI-Sweden-20B

AI-Sweden 20b is a model created by AI Sweden and trained on the Nordic Pile
dataset.

##  3 Types of models evaluated

Since the benchmark contains both open source and closed source, not all
knowledge regarding training of the models is available. However, based on
information regarding the open models some conclusions regarding models can be
drawn.

###  3.1 Instruction tuned models

Instruction tuning involves creating a dataset of instructions (prompts) and
answer for a base trained model to follow. Instruction tuning has been shown
to increase model performance and Llama 8b Instruct and Llama 70b Instruct are
examples of instruction tuned models in this benchmark.

###  3.2 Reinforcement Learning from Human Feedback

Reinforcement Learning from Human Feedback is a step in training where a model
uses human feedback to improve model performance. Most models included in this
benchmark has at some point been trained with RLHF. Notably
Dolphin-2.9-llama3b-8b-flashback is the only model where an attempt has been
made to remove preferences made from RLFH training.

###  3.3 Merged models

Merging[Yadav et al.(2023)Yadav, Tam, Choshen, Raffel, and Bansal] is a
technique where models are combined to form a new model by combining the
weighs inside the neural network directly. This data free training method has
several benefits with perhaps the biggest one being the ability to merge a
model without the need for a GPU. This in combination with the high quality
performance of merge models have made them a common technique for working with
models.

##  4 Next steps

We hope that everyone will help out in our work on improving Swedish Language
models by evaluating models on chatbotarena.se, the home of our benchmark.

Once evaluations are done and we have gathered enough data, we will present a
benchmark of the most performant models for Swedish.

###  4.1 Data

Our benchmark relies on forced binary ranking of two models which gives as
output a preference for an answer over another. This output is useful both as
assessment of model performance but also as a way to create Human Preference
Data that can be used to improve model performance.

###  4.2 How can you help?

The goal of our project is to improve access to high quality language models
in Swedish. The easiest way to help is to simply evaluate models. We also
welcome pull requests and feedback on our
repo111https://github.com/BirgerMoell/SwedishLLMBenchmark.

## References

  * [Anthropic(2023)] Anthropic.  Model card for claude 3.  https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf, 2023\.  Accessed: 2024-05-02.
  * [Bai et al.(2022)Bai, Kadavath, Kundu, Askell, Kernion, Jones, Chen, Goldie, Mirhoseini, McKinnon, Chen, Olsson, Olah, Hernandez, Drain, Ganguli, Li, Tran-Johnson, Perez, Kerr, Mueller, Ladish, Landau, Ndousse, Lukosuite, Lovitt, Sellitto, Elhage, Schiefer, Mercado, DasSarma, Lasenby, Larson, Ringer, Johnston, Kravec, Showk, Fort, Lanham, Telleen-Lawton, Conerly, Henighan, Hume, Bowman, Hatfield-Dodds, Mann, Amodei, Joseph, McCandlish, Brown, and Kaplan] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan.  Constitutional ai: Harmlessness from ai feedback, 2022.
  * [Chiang et al.(2024)Chiang, Zheng, Sheng, Angelopoulos, Li, Li, Zhang, Zhu, Jordan, Gonzalez, and Stoica] Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica.  Chatbot arena: An open platform for evaluating llms by human preference, 2024.
  * [Meta(2023)] Meta.  Llama 3 release blog.  https://ai.meta.com/blog/meta-llama-3/, 2023.  Accessed: 2024-05-02.
  * [Nielsen(2023)] Dan Nielsen.  ScandEval: A benchmark for Scandinavian natural language processing.  In Tanel Alumäe and Mark Fishel, editors, _Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa)_ , pages 185–201, Tórshavn, Faroe Islands, May 2023. University of Tartu Library.  URL https://aclanthology.org/2023.nodalida-1.20.
  * [Yadav et al.(2023)Yadav, Tam, Choshen, Raffel, and Bansal] Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal.  Ties-merging: Resolving interference when merging models, 2023.