# Grey-informed neural network for time-series forecasting

Wanli Xie  Ruibin Zhao  Zhenguo Xu<EMAIL_ADDRESS>Tingting Liang  School of
Communication, Qufu Normal University, Rizhao, China  School of Computer
Science and Information Engineering, Chuzhou Univeristy, Chuzhou, China

###### Abstract

Neural network models have shown outstanding performance and successful
resolutions to complex problems in various fields. However, the majority of
these models are viewed as black-box, requiring a significant amount of data
for development. Consequently, in situations with limited data, constructing
appropriate models becomes challenging due to the lack of transparency and
scarcity of data. To tackle these challenges, this study suggests the
implementation of a grey-informed neural network (GINN). The GINN ensures that
the output of the neural network follows the differential equation model of
the grey system, improving interpretability. Moreover, incorporating prior
knowledge from grey system theory enables traditional neural networks to
effectively handle small data samples. Our proposed model has been observed to
uncover underlying patterns in the real world and produce reliable forecasts
based on empirical data.

###### keywords:

Grey system model, Fractional derivative, Neural network , Time series
forecasting

###### Contents

  1. 1 Introduction
  2. 2 Preliminaries
    1. 2.1 Truncated M-fractional derivative
    2. 2.2 A brief introduction to the grey prediction model
  3. 3 Grey-informed neural network
    1. 3.1 Model expression
  4. 4 Grey-informed neural network with truncated M-fractional difference
    1. 4.1 Definition of the truncated M-fractional accumulation and difference
  5. 5 Verification of FGINN model
  6. 6 Applications and analysis
  7. 7 Conclusion

##  1 Introduction

Due to continuous advancements, neural networks have showcased remarkable
capabilities across various fields, greatly contributing to the progress of
artificial intelligence technologies. Artificial neural network models have
been successfully employed in time series modeling [1], image processing [2],
and natural language processing [3]. In order to effectively tackle complex
practical problems, numerous scholars have made fruitful improvements to
artificial neural network models, resulting in a multitude of significant
findings [4]. Among various research domains, the prediction of time utilizing
artificial neural networks has garnered significant scholarly attention [5].
Present research extensively utilizes the robust modeling capabilities of
neural networks to analyze real-world data and forecast trends within time
series. However, mainstream neural network models require large volumes of
modeling data and lack interpretability. Consequently, numerous scholars have
proposed effective techniques to address these challenges. With the
advancement of grey system theory [6, 7], it is anticipated that this issue
will be effectively addressed. Data modeling in small sample environments can
be effectively addressed through the use of grey system theory. The concept of
a white system denotes a system with complete information, while a grey system
contains a mix of known and unknown information, and a black system is
entirely unknown. Predicting system behavior does not rely on prior data in
the grey system approach. Within grey system theory, certain real-world
development patterns are attributed to grey system dynamics. By employing
specific operators, the underlying laws governing system evolution can be
fully elucidated. This modeling approach is currently experiencing rapid
advancement, distinguishing itself from traditional statistical and fuzzy
mathematical methods in its unique methodology. Figure 1 shows the
relationship between the three systems and the amount of data. Chen et al.
propose the Grey Neural Network (GNN), which combines grey system theory with
neural network methods. The GNN model incorporates a grey layer before the
neural input layer and n white layers after the neural output layer. By
leveraging the strengths of both the grey model and neural network, the GNN
achieves enhanced precision in forecasting. The GNN presents a promising
approach for accurate predictions in various applications [8]. Wu et al.
introduce a novel wave energy forecast model that combines an improved grey
BPNN with a modified ensemble empirical mode decomposition
(MEEMD)-autoregressive integrated moving average (ARIMA) approach. This
integration allows the proposed model to achieve higher accuracy in predicting
wave energy output [9]. Lei et al. propose a new grey forecasting model called
the neural ordinary differential grey model (NODGM), inspired by neural
ordinary differential equations (NODE). Leveraging the latest techniques in
NODE research, the NODGM model represents an innovative approach to grey
forecasting with potential for practical application [10]. Ma et al. adopt the
concept of ”Grey-box” modeling to maximize the benefits of a deterministic
structure. They develop a neural grey system model that combines existing
information with novel neural network techniques. The Levenberg-Marquardt
algorithm is employed to facilitate model training, while Bayesian
regularization is utilized to automatically adjust the regularized parameter,
further enhancing the accuracy and robustness of the proposed model [11]. Xie
et al. present a fractional-order neural grey system model with a three-layer
structure to maximize the advantages of each element. The input of the network
is a fractional-order cumulative sequence, while the output is a predicted
value. By merging these techniques with traditional grey system theory, the
model offers a broader and more comprehensive understanding of the system
under study [12]. As a result of the Richards equation introduced by He et
al., grey modeling has been conducted to simulate traffic parameter
development trends. The method is then fused with the ability to adjust error
feedback as well as complex nonlinear fitting of neural networks in order to
estimate the grey model parameters and forecast traffic volatility [13].

In order to train neural networks, a large amount of data is required, and
neural networks must search the data for patterns completely. Insufficient
data, however, makes it impossible to develop a suitable model. It is
therefore necessary to have a large amount of data in order to train neural
networks using data-driven techniques. As a result of the efforts of
researchers, this problem is being addressed. To guide neural network
training, some scholars have considered using physics laws as prior knowledge
in order to ensure that the inputs and outputs of the model meet certain
physics laws [14]. It should also be noted that such ideas are also well
suited for solving prediction problems. According to Du et al., a new
underwater acoustic field prediction method based on a physics-informed neural
network, called UAFP-PINN, has been developed [15]. Li et al. developed a
physics-informed dynamic mode decomposition method (piDMD), which incorporates
the mass conservation law into a purely data-driven dynamic mode decomposition
method. The spatial and temporal dynamics of solid volume fraction
distribution in bubbling fluidized beds can be quickly predicted [16].
Furthermore, there are also some novel methods that integrate prior knowledge
into neural network models, such as embedding causal relationships into neural
network models to solve thorny engineering issues [17]. In this paper, we
consider embedding the grey system model into the neural network model from
the perspective of systems theory, allowing the neural network to not only
mine patterns from the data, but also ensure that the inputs and outputs of
the neural network meet the patterns described by the grey system.

The research is structured into several distinct sections. The second section
of the paper provides essential background information, including an in-depth
analysis of the grey prediction model. Section 3 presents the GINN model. The
fourth section introduces a modified version of the GINN model utilizing
fractional order difference. The fifth and sixth sections demonstrate the
practical application of the model in real-world scenarios and provide
evidence of its effectiveness. A comprehensive summary of the entire text is
presented in the seventh section as a conclusion.

Nomenclature
---
$\mathcal{T}$ | Sample space | The set of all possible outcomes of an experiment
$\mathcal{D}$ | Label space | The set of possible labels or output values
$\theta$ | Parameters | The weights and biases of the neural network model
$\xi$ | Weighting coefficient | A coefficient to balance the errors of the neural network and grey prediction models
$\alpha$ | Fractional order | The fractional order used in the differential equations of the grey prediction model
$\beta$ | Parameter | A positive parameter in the truncated M-fractional operators
$a,b$ | Indices | Indices for arrays or sequences
$x(t),y(t)$ | Observation sequences |  The original sequence and the output or prediction sequence for time $t$
$f(t)$ | Function |  A function of the values at time $t$
$D$ | Data distribution | The distribution of training data
$L^{ALL}$ | Total error | The sum of the error terms from the neural network model and the grey system model
$L^{NN}$ | Neural network error | The error term of the neural network model
$L^{GM}$ | Grey system error | The error term of the grey system model
Figure 1: The relationship between the white system, grey system, and black
system is intrinsically linked to the quantity of available data. When full
knowledge of a system is attained, data collection becomes unnecessary to
understand its operational principles. However, when only partial information
is available, a grey system is required to detail the operational rules of the
system.

##  2 Preliminaries

Throughout this paper,
${\mathbb{N}_{a}}=\\{a,a+1,a+2,\ldots\\},\mathbb{N}_{a}^{d}=\\{a,a+1,a+2,\ldots,d\\}$,
where $a,d\in\mathbb{R},d-a\in{\mathbb{N}_{1}}$.
$C^{m}([0,+\infty),\mathbb{R})$ represents a set of continuous m-order
differential function from $C^{m}([0,+\infty)$ into $\mathbb{R}$.

###  2.1 Truncated M-fractional derivative

######  Definition 1 ([18]).

The one-parameter truncated Mittag-Leffler function is defined as

${}_{i}\mathbb{E}_{\beta}(z)=\sum_{k=0}^{i}\frac{z^{k}}{\Gamma(\beta k+1)},$
(1)

where $\beta>0$ and $z\in\mathbb{C}$.

######  Definition 2 ([18]).

The truncated M-fractional derivative of the function
$f:[0,\infty)\to\mathbb{R}$ of order $\alpha$ is denoted as follows:

$D_{\beta}^{\alpha}(x):=\lim_{\varepsilon\rightarrow
0}\frac{f\left(x_{i}\mathbb{E}_{\beta}\left(\varepsilon
x^{-\alpha}\right)\right)-f(x)}{\varepsilon}$ (2)

for all $x>0$, ${\rm{0<}}\alpha{\rm{\leq 1}}$ and $\beta>0$.

######  Theorem 1 ([18]).

If f is differentiable and $x>0$, then

$D_{\beta}^{\alpha}(x)=\frac{x^{1-\alpha}}{\Gamma(\beta+1)}\frac{df(x)}{dx},$
(3)

where $\Gamma(\cdot)$ is the Gamma function.

######  Definition 3 ([18]).

Let $a\geq 0$ and $t\geq a$, the truncated $M$-fractional integral of order
$\alpha$ for the function $f$ is formally defined by

$\left(I_{a}^{\alpha,\beta}f\right)(t)=\Gamma(\beta+1)\int_{a}^{t}\frac{f(x)}{x^{1-\alpha}}dx$
(4)

for $\beta>0$ and $0<\alpha<1$.

###  2.2 A brief introduction to the grey prediction model

The purpose of this subsection is to introduce the basic knowledge of grey
prediction models in a brief manner.

######  Definition 4.

Let $f:{\mathbb{N}_{a}}\to\mathbb{R}$, the first-order difference is defined
as

$\nabla f(k):=f(k)-f(k-1)$ (5)

for $k\in{\mathbb{N}_{a+1}}$.

######  Definition 5.

Set $f:{\mathbb{N}_{a+1}}\to\mathbb{R}$ and $b\in{\mathbb{N}_{a}}$, then the
discrete integral of $f$ is defined as

$\nabla^{-1}f(k)=\int\limits_{a}^{b}{f(k)\nabla
k:=\mathop{\sum}\limits_{k=a+1}^{b}f(k)}$ (6)

for $k\in{\mathbb{N}_{a}}$. Specifically, we have

$\nabla^{-1}f(k)=\int_{a}^{a}f(k)\nabla
k:=\mathop{\sum}\limits_{k=a+1}^{a}f(k)=0.$ (7)

Using the aforementioned definition as a starting point, we will now proceed
to present the concept of GM(1,1) [19].

######  Definition 6.

The GM(1,1) model in continuous form can be represented as

$\left\\{{\begin{array}[]{*{20}{l}}\begin{array}[]{l}{y}(t)=\int_{1}^{t}{{x}(\tau)d}\tau,\\\
\frac{d}{{dt}}{y}(t)+{a}{y}(t)={b},\end{array}\\\
{{y}(1)={x}(1).}\end{array}}\right.$ (8)

The solution to equation (8) can be obtained by performing calculations as

${y{}}(t)=\left({{x}(1)-\frac{b}{a}}\right){e^{-a(t-1)}}+\frac{b}{a}.$ (9)

In order to estimate parameters of the GM(1,1) model, it is necessary to
discretize it.

######  Definition 7.

Let ${{X}^{(0)}}=\left\\{{{x}(1),{x}(2),\cdots,{x}(n)}\right\\}$ is the
original time series, then the GM(1,1) model in discrete form can be defined
as

$\left\\{{\begin{array}[]{*{20}{l}}{\begin{array}[]{*{20}{l}}{{\nabla^{-1}}x(k)=\sum\limits_{\tau=1}^{k}{x(\tau)},}\\\
{x(k)+a{\nabla^{-1}}x(k)=b,}\end{array}}\\\
{{\nabla^{-1}}x(1)=x(1)}\end{array}}\right.$ (10)

for $k\in\mathbb{N}_{1}^{n}$.

Using the least squares method, the parameters of the GM(1,1) model can be
determined as follows:

${[\hat{a},\hat{b}]^{T}}={\left({{B^{T}}B}\right)^{-1}}{B^{T}}Y,$ (11)

where

$B=\left[{\begin{array}[]{*{20}{c}}{-\frac{1}{2}\left({{\nabla^{-1}}x(2)+{\nabla^{-1}}x(1)}\right)}&1\\\
{-\frac{1}{2}\left({{\nabla^{-1}}x(3)+{\nabla^{-1}}x(2)}\right)}&1\\\
\vdots&\vdots\\\
{-\frac{1}{2}\left({{\nabla^{-1}}x(n)+{\nabla^{-1}}x(n-1)}\right)}&1\end{array}}\right],$
(12) $Y=\left[{\begin{array}[]{*{20}{c}}{{\nabla^{-1}}(2)-{\nabla^{-1}}(1)}\\\
{{\nabla^{-1}}(3)-{\nabla^{-1}}(2)}\\\ \vdots\\\
{{\nabla^{-1}}(n)-{\nabla^{-1}}(n-1)}\end{array}}\right].$ (13)

Based on the estimated parameters and the discrete response function, the
predicted values of the series can be calculated as follows:

${\nabla^{-1}\hat{x}}(k)=\left({{x}(1)-\frac{{\hat{b}}}{{\hat{a}}}}\right){e^{-\hat{a}(k-1)}}+\frac{{\hat{b}}}{{\hat{a}}},k\in\mathbb{N}_{1}^{m}.$
(14)

As a result, the restored values can be written as follows:

${\nabla^{-1}\hat{x}}(k)={\nabla^{-1}\hat{x}}(k)-{\nabla^{-1}\hat{x}}(k-1),k\in\mathbb{N}_{2}^{m}.$
(15)

######  Lemma 1 ([20]).

Set $a,b\in C(\left[t_{0},T\right],\mathbb{R})$, and $x\in
C^{1}\left(\left[t_{0},T\right],\mathbb{R}\right)$ satisfies

$\left\\{{\begin{array}[]{*{20}{l}}{\frac{{{\rm{d}}x(t)}}{{dt}}\leq
a(t)x(t)+u(t),t\in\left[{{t_{0}},T}\right],}\\\
{x\left({{t_{0}}}\right)\leq{x_{0}}.}\end{array}}\right.$ (16)

Then

$x(t)\leq
x_{0}\exp\left(\int_{t_{0}}^{t}a(\rho)d\rho\right)+\int_{t_{0}}^{t}\exp\left(\int_{\rho}^{t}a(r)dr\right)u(\rho)d\rho,\quad
t\in\left[t_{0},T\right].$ (17)

##  3 Grey-informed neural network

###  3.1 Model expression

Let $\mathcal{X}$ denote the sample space and $\mathcal{Y}$ the label space.
The distribution of training data on $\mathcal{X}\times\mathcal{Y}{\tiny}$ is
denoted as $D$, with the training dataset
$S=\left\\{\left(x_{i},y_{i}\right)\right\\}_{i=1}^{n}$ comprising $n$
independent data points sampled from $D$. The model parameters are denoted by
$\theta\in\Theta\subseteq\mathbb{R}^{d}$. The open ball of radius $\rho>0$
centered at $\theta$ in Euclidean space is noted as $B(\theta,\rho)$, defined
as
$B(\theta,\rho)=\left\\{\theta^{\prime}:\left\|\theta-\theta^{\prime}\right\|<\rho\right\\}$.
The L2 norm is represented as $\|\cdot\|$. The loss function per data point,
denoted as
$\ell:\Theta\times\mathcal{X}\times\mathcal{Y}\rightarrow\mathbb{R}$, is
labeled as $\ell$. The empirical loss function $\hat{L}(\theta)$, is
calculated as $\sum_{i=1}^{n}\ell\left(\theta,x_{i},y_{i}\right)$. The
gradient matrix of the function $\hat{L}(\cdot)$ at point $\theta$ are denoted
as $\nabla\hat{L}(\theta)$. Furthermore, $L^{\text{oracle }}(\theta)$ is used
in this paper to represent the oracle loss function. Building upon the
aforementioned definition, we introduce a novel neural network error function,
which is outlined as follows:

${L^{ALL}}(\theta)=\hat{L}(\theta)+\xi{L^{GM}}(\theta)$ (18)

for $\xi\in\left[{0,1}\right]$, where ${L^{ALL}}(\theta)$ denotes the total
error of the neural network, while $\hat{L}(\theta)$ represents the network’s
specific error. Additionally, ${L^{GM}}(\theta)$ signifies the error of the
grey system, with $\xi$ serving as the weighting coefficient. The neural
network model constructed with such an error function is called the GINN
model. The error function of our proposed GINN model departs significantly
from that of traditional neural networks. Our novel error function is composed
of two unique components: the first component represents the classic error
term of neural networks, while the second component incorporates differential
equations derived from grey systems. In practical computations, we will
utilize a differential form for efficient computation. This dual-component
approach serves a dual purpose: firstly, it allows neural networks to adhere
to a data-driven mechanism, facilitating the extraction of objective patterns
from data; secondly, it ensures that neural networks conform to the dynamic
laws outlined by grey systems. By integrating the dynamic laws articulated by
differential equations, our model introduces new prior knowledge into neural
networks, enabling them to effectively model with less data. In the neural
network framework, $\hat{L}(\theta)$ can be written as $L^{NN}(\theta)$, so
equation (18) can be further rewritten as

${L^{ALL}}(\theta)={L^{NN}}(\theta)+\xi{L^{GM}}(\theta).$ (19)

If we use the Mean Squared Error (MSE) function [21] as the standard for
calculating error, equation (19) can be further modified as

$MSE=MSE_{NN}+\xi MSE_{GM},$ (20)

where

${{\rm{MS}}{{\rm{E}}_{{\rm{NN}}}}{\rm{=}}\frac{1}{n}\sum\limits_{i=1}^{n}{{{\left|{{{y}}_{\rm{i}}{\rm{-}}{{{f}}_{\rm{i}}}}\right|}^{\rm{2}}}},}$
(21)
${{\rm{MS}}{{\rm{E}}_{GM}}{\rm{=}}\frac{1}{n}\sum\limits_{i=1}^{n}{{{\left|{{{y}}_{\rm{i}}{\rm{-}}{{{g}}_{\rm{i}}}}\right|}^{\rm{2}}}}.}$
(22)

In the context of a given multivariate time series input, i.e., a sliding
window $X_{i}=\left[x_{i-T+1},\ldots,x_{i}\right]\in\mathbb{R}^{N\times T}$,
where at time $i$, the number of variables in the sequence is $N$ and the size
of the sliding window is $T$, with $x_{i}\in\mathbb{R}^{N}$ representing the
corresponding $N$-dimensional multivariate values at time $i$. Therefore, the
task of multivariate time series forecasting is to predict the next $\tau$
timestamps
$Y_{i}=\left[x_{i+1},\ldots,x_{i+\tau}\right]\in\mathbb{R}^{N\times\tau}$
based on the historical $T$ observations. So the training set for time series
problems is represented in this paper as
$S=\left\\{{\left({{{\left[{{x_{i-T+1}},\ldots,{x_{i}}}\right]}},{Y_{i}}}\right)}\right\\}_{i=1}^{n}$.
Specifically, when $\tau=1$, we have
$S=\left\\{{\left({{{\left[{{x_{i-T+1}},\ldots,{x_{i}}}\right]}},{x_{i+1}}}\right)}\right\\}_{i=1}^{n}$.

Figure 2: Schematic diagram of the network structure of GINN. The error in the
proposed neural network model can be divided into two components. The first
component stems from the disparity between the predicted values and the actual
values within the neural network. The second component pertains to the error
within the grey system model.

##  4 Grey-informed neural network with truncated M-fractional difference

Based on the GINN model, we introduce a novel model known as fractional grey
informed model in this section. This model is developed using a innovative
fractional order accumulation operator.

###  4.1 Definition of the truncated M-fractional accumulation and difference

######  Definition 8.

Considering an arbitrary function $f:\mathbb{N}_{a}\rightarrow\mathbb{R}$, we
define the truncated M-fractional difference (tM-D) in the following manner:

${\nabla^{\alpha}}f(k):=\frac{{{k^{1-\alpha}}}}{{\Gamma(\beta+1)}}\left({f(k)-f(k-1)}\right),k\in{\mathbb{N}_{a+1}}.$
(23)

######  Definition 9.

Set $f:\mathbb{N}_{a}\rightarrow\mathbb{R}$, the truncated M-fractional
accumulation (tM-A) can be defined as

${\nabla^{-\alpha}}f(k):=\Gamma(\beta+1)\int_{a+1}^{k}{\frac{{f(k)}}{{{k^{1-\alpha}}}}}\nabla
t:=\Gamma(\beta+1)\sum\limits_{t=a}^{k}{\frac{{f(k)}}{{{k^{1-\alpha}}}}}$ (24)

for $k\in{\mathbb{N}_{a}}$, where $g\leq h$ are in $\mathbb{N}_{a}$.

######  Property 1.

Let
$F(t):=\Gamma(\beta+1)\int_{b+1}^{t}{\frac{{f(s)}}{{{x^{1-\alpha}}}}}\nabla
s$, for $t\in\mathbb{N}_{b}^{a}$, then we have $\nabla^{\alpha}F(t)=f(t)$,
where $t\in\mathbb{N}_{b+1}^{a}$.

###### Proof.

Set $F(t)=\int_{b}^{t}f(s)\nabla s,\quad t\in\mathbb{N}_{b}^{a}$, then

$\displaystyle\nabla^{\alpha}F(t)=\nabla^{\alpha}\left(\Gamma(\beta+1)\int_{a+1}^{t}\frac{f(s)}{s^{1-\alpha}}\nabla
s\right)$ (25)
$\displaystyle=\nabla^{\alpha}\left(\Gamma(\beta+1)\sum_{s=a}^{t}\frac{f(s)}{s^{1-\alpha}}\right)$
$\displaystyle=\frac{k^{1-\alpha}}{\Gamma(\beta+1)}\left(\Gamma(\beta+1)\sum_{s=a}^{t}\frac{f(s)}{s^{1-\alpha}}-\Gamma(\beta+1)\sum_{s=a}^{t-1}\frac{f(s)}{s^{1-\alpha}}\right)$
$\displaystyle=f(t)$

for $t\in\mathbb{N}_{b+1}^{a}.$ ∎

By integrating new accumulation and difference operators, we propose a novel
fractional order grey model, denoted as tM-FGM (1,1), which incorporates the
fractional order integration operator. The primary objective of this study is
to elucidate the fundamental architecture of the model. Parameter estimation
and prediction methodologies employed in tM-FGM (1,1) are rooted in the GM
(1,1) theory. Our aim is to establish a comprehensive framework for the
application of this model.

######  Definition 10.

The proposed grey prediction mode with truncated M-fractional integral in
continuous form can be represented as

$\left\\{{\begin{array}[]{*{20}{c}}{y(t)=\Gamma(\beta+1)\int_{a}^{t}{\frac{{x(s)}}{{{s^{1-\alpha}}}}}ds,}\\\
{y(t)+ay(t)=b,}\\\ {y(1)=x(1),}\end{array}}\right.$ (26)

where $y(t)=D_{\beta}^{\alpha}I_{a}^{\alpha,\beta}y(t)$.

######  Definition 11.

For $k\in\mathbb{N}_{1}^{n}$, the proposed grey prediction model in discrete
form incorporating tM-A is represented as

$\left\\{{\begin{array}[]{*{20}{c}}{y(k)=\Gamma(\beta+1)\int_{b+1}^{\rm{k}}{\frac{{x(\lambda)}}{{{\lambda^{1-\alpha}}}}}\nabla\lambda,}\\\
{y(k)+ay(k)=b,}\\\ {y(1)=x(1).}\end{array}}\right.$ (27)

where ${\nabla^{\alpha}}{\nabla^{-\alpha}}y(k)$.

Our next step is to present a novel set of fractional Gronwall inequality to
analyze the characteristics of solutions in grey systems according to the
definition of fractional integration.

######  Theorem 2.

Consider a non-negative, monotonically non-decreasing function $f(t)$ defined
on the interval $[a,b]$, where $t\in[a,b]$. Let $g(t)$ be a non-negative
function that satisfies

$x(t)\leq
g(t)+\Gamma(\beta+1)\int_{a}^{t}f(\tau)x(\tau){\tau^{\alpha-1}}d\tau,$ (28)

then

$\displaystyle x(t)\leq
g(t)+\int_{a}^{t}g(s)f(s)\exp[\Gamma(\beta+1)\int_{s}^{t}f(\tau){\tau^{\alpha-1}}d\tau]{d_{\alpha}}s$
(29)

for $t\in[0,+b]$, where ${d_{\alpha}}{\rm{s=}}{s^{\alpha-1}}ds$.

###### Proof.

Set
$G(t)=g(t)+\Gamma(\beta+1)\int_{a}^{t}f(\tau)x(\tau){\tau^{\alpha-1}}d\tau$,
then $G(a)=g(a)$ and $x(t)\leq G(t)$. Upon calculating the fractional
derivative of $G(t)$ with respect to time on both sides, we obtain

$\displaystyle D_{\beta}^{\alpha}G(t)$
$\displaystyle=D_{\beta}^{\alpha}g(t)+D_{\beta}^{\alpha}\Gamma(\beta+1)\int_{a}^{t}f(\tau)x(\tau)\tau^{\alpha-1}d\tau$
(30) $\displaystyle=D_{\beta}^{\alpha}g(t)+f(t)x(t).$

Based on the inequality $x(t)\leq G(t)$, it can be inferred that

$D_{\beta}^{\alpha}G(t)\leq D_{\beta}^{\alpha}g(t)+f(t)G(t).$ (31)

By applying equation (31), we can multiply both sides by
$\exp\left[{-\Gamma(\beta+1)\int_{a}^{t}f(\tau){\tau^{\alpha-1}}d\tau}\right]$,
one has

$\displaystyle\exp\left[-\Gamma(\beta+1)\int_{a}^{t}f(\tau)(\tau-a)^{\alpha-1}ds\right]D_{\beta}^{\alpha}G(t)$
(32)
$\displaystyle\leq\exp\left[-\Gamma(\beta+1)\int_{a}^{t}f(\tau)(\tau-a)^{\alpha-1}ds\right]\left(D_{\beta}^{\alpha}g(t)+f(t)G(t)\right).$

An additional level of organization can result in

$\displaystyle\exp\left[{-\Gamma(\beta+1)\int_{a}^{t}f(\tau)\tau^{\alpha-1}ds}D_{\beta}^{\alpha}G(t)\right]$
(33)
$\displaystyle-\exp\left[{-\Gamma(\beta+1)\int_{a}^{t}f(\tau)\tau^{\alpha-1}ds}f(t)G(t)\right]$
$\displaystyle=D_{\beta}^{\alpha}\left\\{G(t)\exp\left[{-\Gamma(\beta+1)\int_{a}^{t}f(\tau)\tau^{\alpha-1}d\tau}\right]\right\\}$
$\displaystyle\leq\exp\left[{-\Gamma(\beta+1)\int_{a}^{t}f(\tau)\tau^{\alpha-1}d\tau}D_{\beta}^{\alpha}g(t)\right].$

The properties of fractional-order integration are used to obtain

$\displaystyle\left.G(t)\exp\left[-\Gamma(\beta+1)\int_{a}^{t}f(\tau)\tau^{\alpha-1}d\tau\right]\right|_{a}^{t}$
(34)
$\displaystyle=G(t)\exp\left[-\Gamma(\beta+1)\int_{a}^{t}f(\tau)\tau^{\alpha-1}d\tau\right]-G(a)$
$\displaystyle=G(t)\exp\left[-\Gamma(\beta+1)\int_{a}^{t}f(\tau)\tau^{\alpha-1}d\tau\right]-g(a)$

Applying fractional-order integration to both sides of equation (33), we have

$\displaystyle
G(t)\exp\left[-\Gamma(\beta+1)\int_{a}^{t}f(\tau)\tau^{\alpha-1}d\tau\right]$
(35) $\displaystyle\leq
g(a)+\int_{a}^{t}\exp\left[-\Gamma(\beta+1)\int_{a}^{s}f(\tau)\tau^{\alpha-1}d\tau\right]D_{\beta}^{\alpha}g(s)d_{\alpha}s$
$\displaystyle=g(a)+\left.\exp\left[-\Gamma(\beta+1)\int_{a}^{s}f(\tau)\tau^{\alpha-1}d\tau\right]g(s)\right|_{a}^{t}$
$\displaystyle-\int_{a}^{t}g(s)D_{\beta}^{\alpha}-\Gamma(\beta+1)\int_{a}^{s}f(\tau)\tau^{\alpha-1}d\tau
d_{\alpha}s$
$\displaystyle=\exp\left[-\Gamma(\beta+1)\int_{a}^{t}f(\tau)\tau^{\alpha-1}d\tau\right]g(t)$
$\displaystyle-\int_{a}^{t}g(s)D_{a}^{\alpha}\exp\left[-\Gamma(\beta+1)\int_{a}^{s}f(\tau)\tau^{\alpha-1}d\tau\right]d_{\alpha}s$
$\displaystyle=\exp\left[-\Gamma(\beta+1)\int_{a}^{t}f(\tau)\tau^{\alpha-1}d\tau\right]g(t)$
$\displaystyle+\int_{a}^{t}g(s)f(s)\exp\left[-\Gamma(\beta+1)\int_{a}^{s}f(\tau)\tau^{\alpha-1}d\tau\right]d_{\alpha}s.$

Multiplying both sides by
$\exp[\Gamma(\beta+1)\int_{a}^{t}f(\tau){\tau^{\theta-1}}d\tau]$, we have

$\displaystyle G(t)\leq
g(t)+\exp\left[\Gamma(\beta+1)\int_{a}^{t}f(\tau)\tau^{\theta-1}d\tau\right]$
(36)
$\displaystyle\quad\quad\times\int_{a}^{t}g(s)f(s)\exp\left[-\Gamma(\beta+1)\int_{a}^{s}f(\tau)\tau^{\theta-1}d\tau\right]d_{\alpha}s.$

Based on condition $x(t)\leq G(t)$, one has

$\displaystyle x(t)\leq
g(t)+\exp[\Gamma(\beta+1)\int_{a}^{t}f(\tau){\tau^{\theta-1}}d\tau]$ (37)
$\displaystyle\quad\quad\times\int_{a}^{t}{g(s)f(s)\exp[-\Gamma(\beta+1)\int_{a}^{s}f(\tau){\tau^{\theta-1}}d\tau]}{d_{\alpha}}s.$

Our organization has been further enhanced by

$x(t)\leq
g(t)+\int_{a}^{t}{g(s)f(s)\exp\left\\{{\Gamma(\beta+1)\Xi}\right\\}}{d_{\alpha}}s,$
(38)

where

$\Xi=\int_{a}^{t}f(\tau){(\tau-a)^{\theta-1}}d\tau-\int_{a}^{s}f(\tau){(\tau-a)^{\theta-1}}d\tau.$
(39)

Based on the properties of the definite integral, the derivation of equation
(29) can be obtained through proper collation. ∎

######  Theorem 3.

If ${y}(t)$ is viewed as the accumulated generating series for original
sequence, $\hat{y}(t)$ is the fitted value of ${y}(t)$ and $f(t)=1$. If

$\hat{y}(t)-y(t)=\phi(t)>0,$ (40)

then

$x(t)\leq\varphi(t)+\int_{a}^{t}\varphi(s)\exp[\Gamma(\beta+1)\int_{s}^{t}{{\tau^{\alpha-1}}d\tau}]{d_{\alpha}}s.$
(41)

###### Proof.

Calculating the derivative of equation (40) on both sides yields the following
result:

$\hat{x}(t)-x(t)=\frac{t^{1-\alpha}}{\Gamma(\beta+1)}\phi^{\prime}(t).$ (42)

Then from (24) one has

$\hat{x}(t)\leq\varphi(t)+\Gamma(\beta+1)\int_{a}^{t}{x(\tau){\tau^{\alpha-1}}d\tau,}$
(43)

where $\frac{t^{1-\alpha}}{\Gamma(\beta+1)}\phi^{\prime}(t)=\varphi(t)$. It
follows from Theorem (29), one has

$x(t)\leq\varphi(t)+\int_{a}^{t}\varphi(s)\exp[\Gamma(\beta+1)\int_{s}^{t}{{\tau^{\alpha-1}}d\tau}]{d_{\alpha}}s.$
(44)

The proof of Theorem 41 is finished. ∎

This analysis indicates that the neural network model based on the tM$-$GM
(1,1) model can be described as fractional-order grey-informed neural network
(FGINN), and the error function of this neural network is as follows:

${L^{ALL}}(\theta)={L^{NN}}(\theta)+\xi{L^{tM-FGM}}(\theta).$ (45)

A schematic diagram illustrating the error fusion method between neural
networks and grey system models can be found in Figure 2. The general grey
information neural network modeling process is delineated in Algorithm 1. A
flow chart illustrating the process from data collection to prediction can be
found in Figure 3.

Figure 3: Framework of main research content: a comprehensive analysis.

Input: Training data $X$, Parameters of Grey Prediction Model $\theta$

Output: Predicted values $\hat{Y}$

1 Initialize weights and biases in neural network;

2 Initialize learning rate $\eta$, iteration number $n_{iter}$;

3 for  _$i=1$ to $n_{iter}$_ do

4 Determine the prediction $\hat{Y}$ using the neural network with the current
weights and biases;

5 Computer loss $L_{MSE}$ using Mean Squared Error between predicted $\hat{Y}$
and true $Y$;

6 Calculate loss $L_{Grey}$ using Grey Prediction Model with parameters
$\theta$;

7 Obtain the total loss $L=L_{MSE}+L_{Grey}$;

8 Update weights and biases in neural network using gradient descent on $L$;

9

10return Predicted values $\hat{Y}$;

Algorithm 1 Grey-informed neural network

##  5 Verification of FGINN model

In this section, we examine six practical examples in order to verify the
effectiveness of the model. Table 1 provides details of the relevant data. For
the grey prediction model, the error term is not calculated using the square
term, in order to avoid excessive error and difficulty in convergence. In this
study, we employed the mean absolute error (MAE) as our error metric, which is
calculated using the formula
$M{\rm{A}}{E_{GM}}=\frac{1}{n}\sum\limits_{i=1}^{n}|{y_{i}}-{g_{i}}|$, with
the weighting coefficient set to 0.1.

Table 1: Information regarding the datasets obtained from the time series data library. No. | Data Title
---|---
1 | Count of county hospitals (units)
2 | Average number of health technical personnel per county hospital (persons)
3 | Average number of beds per county maternal and child health institution (beds)
4 | Average number of health technical personnel per county maternal and child health institution (persons)
5 | Number of township health centers (units)
6 | Proportion of rural doctors (%)
Note: The time dimension of the data is from 2003 to 2022, with data from 2003
to 2018 used to fit the model and data from 2019 to 2022 used to test the
performance of the data.

Utilizing a dataset obtained from the time series data library
111http://stjj.guizhou.gov.cn/ spanning the years 2003 to 2022, a comparative
analysis was conducted on various indices for models including FGINN, GINN,
MLP [22], CFGM [23], FGM [24], FHGM [25], GM [26], and DGM [27]. The relevant
experimental results are shown in Table 2. The performance of the model was
assessed through a comprehensive set of metrics, including Mean Absolute
Percentage Error (MAPE), Mean Squared Error (MSE), Mean Absolute Error (MAE),
and Root Mean Squared Error (RMSE). In most cases, GINN proved to be superior
to grey prediction models or artificial neural network based on the
experimental results. However, GINN’s predictive performance may sometimes be
lower than that of traditional models. In light of this, it is evident that
GINN still has room for improvement. Based on the experimental findings, the
FGINN model yielded the highest accuracy, achieving the lowest values across
all evaluated metrics. This result underscores the model’s precision for this
particular test. This experiments further reinforced FGINN’s dominance,
consistently outperforming other models in terms of predictive accuracy across
different datasets. A detailed analysis of the results revealed that FGINN
consistently outperformed GINN across all six experiments, showcasing lower
error values in MAPE, MSE, MAE, and RMSE. Notably, FGINN displayed a
considerable improvement in predictive accuracy compared to GINN, as evidenced
by consistently lower error values in all metrics across various datasets. For
instance, in the third dataset, FGINN achieved a MAPE of 4.28894% compared to
GINN’s 4.758044%, highlighting FGINN’s superior forecasting capabilities. The
comprehensive comparison of error metrics underscores FGINN’s robustness as a
superior forecasting model compared to GINN. These findings emphasize the
significance of advanced modeling techniques, like FGINN, in enhancing
predictive accuracy and reliability in time series forecasting applications.
Researchers and practitioners can leverage these insights to enhance
forecasting methodologies and achieve more precise predictions in diverse
domains.

Table 2: Validation results of FGINN, GINN, GM, DGM, CFGM, FGM, FHGM and MLP with the benchmark data sets. Number |  Indices | FGINN | GINN | MLP | CFGM | FGM | FHGM | GM | DGM
---|---|---|---|---|---|---|---|---|---
1 |  MAPE |  0.62327 |  0.737393 |  0.786645 |  3.3247 |  3.1655 |  3.4552 |  4.8882 |  4.8484
|  MSE |  0.299435 |  0.28508 |  0.420492 |  4.6776 |  4.2505 |  5.0529 |  10.11 |  9.9481
|  MAE |  0.39554 |  0.470469 |  0.500042 |  2.1316 |  2.0294 |  2.2151 |  3.1332 |  3.1077
|  RMSE |  0.547206 |  0.53393 |  0.648453 |  2.1628 |  2.0617 |  2.2479 |  3.1796 |  3.1541
2 |  MAPE |  3.68056 |  3.70855 |  3.771024 |  14.177 |  17.517 |  14.177 |  14.177 |  14.563
|  MSE |  599.515 |  607.2719 |  625.7444 |  10954 |  14927 |  10954 |  10954 |  11390
|  MAE |  22.1485 |  22.31497 |  22.68692 |  86.791 |  106.57 |  86.791 |  86.791 |  89.085
|  RMSE |  24.485 |  24.64289 |  25.01488 |  104.66 |  122.18 |  104.66 |  104.66 |  106.72
3 |  MAPE |  4.28894 |  4.758044 |  5.53546 |  14.945 |  15.722 |  14.945 |  14.945 |  15.29
|  MSE |  14.226 |  16.36143 |  20.43448 |  147.36 |  160.61 |  147.36 |  147.36 |  153.69
|  MAE |  3.05837 |  3.406374 |  3.984379 |  11.451 |  12.025 |  11.451 |  11.451 |  11.711
|  RMSE |  3.77174 |  4.044927 |  4.520451 |  12.139 |  12.673 |  12.139 |  12.139 |  12.397
4 |  MAPE |  3.1733 |  3.224458 |  3.407433 |  13.998 |  5.2095 |  13.998 |  13.998 |  13.767
|  MSE |  33.7236 |  34.3879 |  37.52559 |  221.79 |  69.59 |  221.79 |  221.79 |  215.29
|  MAE |  3.65579 |  3.703505 |  3.879494 |  14.432 |  5.8065 |  14.432 |  14.432 |  14.203
|  RMSE |  5.80721 |  5.864119 |  6.125814 |  14.893 |  8.3421 |  14.893 |  14.893 |  14.673
5 |  MAPE |  0.7304 |  0.73226 |  0.731297 |  2.4883 |  2.4977 |  2.5206 |  3.1693 |  3.1678
|  MSE |  156.3734 |  155.9875 |  157.3502 |  1202.9 |  1210.9 |  1232.1 |  1914.1 |  1912.4
|  MAE |  10.0261 |  10.05231 |  10.03806 |  34.168 |  34.297 |  34.611 |  43.521 |  43.501
|  RMSE |  12.50493 |  12.4895 |  12.54393 |  34.683 |  34.799 |  35.102 |  43.75 |  43.73
6 |  MAPE |  8.27231 |  10.18479 |  9.245461 |  23.848 |  22.643 |  23.848 |  23.848 |  23.89
|  MSE |  104.757 |  135.5512 |  127.926 |  540.12 |  489.77 |  540.12 |  540.12 |  541.85
|  MAE |  7.44252 |  9.138746 |  8.332684 |  21.923 |  20.828 |  21.923 |  21.923 |  21.962
|  RMSE |  10.2351 |  11.64264 |  11.31044 |  23.241 |  22.131 |  23.241 |  23.241 |  23.278
Note: In the table, the configuration for GINN, FGINN, and MLP models includes
10 hidden layers each, with T = 2 and MSE as the error function. The learning
rate is fixed at 0.001, and the models are trained for 2000 iterations. The
orders FGM, FHGM, CFGM, etc. are determined using the PSO algorithm.

In summary, the recently introduced FGINN and GINN models exhibit distinct
advantages in real-world modeling scenarios, boasting robust generalization
capabilities and adeptly handling time series predictions within small-sample
contexts. Our framework synergistically merges the nonlinear approximation
prowess of neural networks with the strengths of grey prediction models,
tailored for small-sample prediction tasks. Empirical comparisons revealed
that FGINN outperforms GINN, suggesting that the integration of fractional
calculus is efficacious and capable of a spectrum of time series prediction
challenges.

##  6 Applications and analysis

This study aims to assess the forecasting capabilities of the FGINN and GINN
models in predicting electricity consumption in four regions of China and to
compare their performance against established machine learning models. The
regions under consideration include Tianjin, Liaoning, Fujian and Gansu, with
data spanning from 2000 to 2019 (measured in billions of kilowatt-hours).
Table 3 presents the experimental results. The dataset was divided into
training and testing sets for model training and performance evaluation,
respectively. Various evaluation metrics, including MAPE, MSE, MAE, and RMSE,
were employed to gauge the accuracy and reliability of the models in
predicting electricity consumption. The FGINN model consistently outperformed
all competing models in terms of accuracy across all regions and metrics, with
the lowest error values observed. This underscores the superior predictive
capabilities of the FGINN model in forecasting electricity consumption
patterns. Although the GINN model exhibited favorable performance in certain
regions and metrics, it did not consistently match the accuracy levels
achieved by the FGINN model. Specific examination of the data illustrates the
supremacy of the FGINN model in delivering the lowest MAPE values across all
regions, reflecting its precision in predicting percentage errors. For
example, in Tianjin, the FGINN model achieved a remarkable MAPE of 3.78439%,
contrasting with the GINN model’s MAPE of 10.0289% and those of other
comparator models. Notably, the FGINN model exhibited the most favorable MSE
outcomes, with minimal values recorded across all regions. In Tianjin, the
FGINN model yielded an MSE of 1306.26, significantly surpassing the GINN
model’s MSE of 7567.113 and the outputs of alternative models. Further
validation of the FGINN model’s performance reveals its dominance in MAE
values, indicating closest proximity to actual consumption data. In Tianjin,
the FGINN model displayed an MAE of 31.193 in contrast with the GINN model’s
MAE of 82.83217. Evaluation of RMSE values underscored the FGINN model’s
consistent accuracy in predicting residuals’ standard error, particularly
evidenced by the lowest values recorded across most regions. These results
ascertain the FGINN model as a robust forecasting tool for electricity
consumption, securing a prominent position in achieving excellent accuracy and
precision compared to competitor models. While the GINN model demonstrates
promise, its potential can be further refined to approach the consistently
superior performance exhibited by the FGINN model. These findings carry
substantial implications for enhancing decision-making processes concerning
energy utilization within the examined regions of China.

Table 3: Predicted errors of eight competing models in four regions. Region |  Indices |  FGINN |  GINN |  MLP |  CFGM |  FGM |  FHGM |  GM |  DGM
---|---|---|---|---|---|---|---|---|---
Tianjin |  MAPE |  3.78439 |  10.0289 |  8.46356 |  22.697 |  11.641 |  23.241 |  29.55 |  29.617
|  MSE |  1306.26 |  7567.113 |  5416.489 |  38891 |  10155 |  40856 |  66663 |  66934
|  MAE |  31.193 |  82.83217 |  69.8782 |  191.17 |  97.879 |  195.78 |  249.13 |  249.69
|  RMSE |  36.1422 |  86.98916 |  73.5968 |  197.21 |  100.77 |  202.13 |  258.19 |  258.72
Liaoning |  MAPE |  1.72701 |  1.767999 |  1.730181 |  14.399 |  3.4823 |  14.78 |  19.517 |  19.514
|  MSE |  2208.007 |  2111.49 |  2196.884 |  1.03E+05 |  7571.6 |  1.09E+05 |  1.93E+05 |  1.93E+05
|  MAE |  38.3682 |  39.34583 |  38.44592 |  319.46 |  74.133 |  328.12 |  435.04 |  434.94
|  RMSE |  46.98943 |  45.9509 |  46.87092 |  320.54 |  87.015 |  329.42 |  439.45 |  439.3
Fujian |  MAPE |  1.60132 |  2.080047 |  1.658052 |  9.648 |  3.7342 |  11.798 |  19.008 |  19.179
|  MSE |  2537.6 |  2778.401 |  2365.615 |  48002 |  7336 |  72759 |  1.93E+05 |  1.97E+05
|  MAE |  36.3702 |  45.88663 |  37.66058 |  213.65 |  81.57 |  261.99 |  423.82 |  427.58
|  RMSE |  50.3746 |  52.71054 |  48.63759 |  219.09 |  85.65 |  269.74 |  439.69 |  443.38
Gansu |  MAPE |  5.89752 |  5.903156 |  5.97287 |  19.689 |  14.038 |  20.15 |  27.423 |  27.55
|  MSE |  5277.98 |  5291.477 |  5506.729 |  57923 |  29292 |  60805 |  1.15E+05 |  1.16E+05
|  MAE |  70.711 |  70.78107 |  71.67993 |  236.5 |  167.66 |  242.19 |  331.27 |  332.79
|  RMSE |  72.6497 |  72.74254 |  74.20734 |  240.67 |  171.15 |  246.59 |  338.93 |  340.42
Note:  In the table, each of the GINN, FGINN, and MLP models has 10 hidden
layers, with T = 2 and MSE as the error function. Models are trained for 2000
iterations at a learning rate of 0.001. Based on the PSO algorithm, the orders
FGM, FHGM, CFGM, etc. are determined.
M1M2M3M4M5M6M7M8$0$$10$$20$$30$$40$Performance
(APE)TianjinM1M2M3M4M5M6M7M8$0$$10$$20$LiaoningM1M2M3M4M5M6M7M8$0$$10$$20$$30$FujianM1M2M3M4M5M6M7M8$0$$10$$20$$30$$40$Gansu
Figure 4: An illustration of each point error for all models is shown in a box
plot. These include ”M1”, ”M2”, ”M3”, ”M4”, ”M5”, ”M6”, ”M7”, and ”M8” which
represent FGINN, GINN, MLP, CFGM, FGM, FHGM, GM, and DGM, respectively.

##  7 Conclusion

As a result, the use of grey-informed neural networks addresses the challenges
associated with black-box neural network models in scenarios with limited
sample sizes. GINN enhances interpretability and the ability of traditional
neural networks to handle small samples by incorporating the differential
equation model of grey systems. Its effectiveness in mitigating data scarcity
issues in neural network modeling can be demonstrated by its use of potential
underlying laws in the real world to make reasonable predictions based on
actual data. There are a number of areas that warrant further exploration in
this study, even though it presents a novel approach for developing a grey
neural network. Firstly, determining the optimal ratio of error terms in
neural networks and grey prediction models is a promising research direction.
In this paper, we are solely concerned with univariate prediction models,
whereas grey models have distinct characteristics that are capable of
capturing a wide range of patterns. Future research should focus on selecting
an appropriate grey prediction model tailored to specific real-world problems.

Declaration of competing interest

The authors declare that they have no known competing financial interests or
personal relationships that could have appeared to influence the work reported
in this paper.

Data and code availability

The code is available in
https://github.com/Sunflowersandshells/GINN/tree/master.

Acknowledgments

This work is supported by the Science and Technology Plan Project in Chuzhou
(No. 2021ZD016), the Research Projects of the Chuzhou University (No.
2022XJZD14), and the Anhui Province’s University Research Projects (No.
2023AH040216).

## References

  * [1] H. Ismail Fawaz, G. Forestier, J. Weber, L. Idoumghar, P.-A. Muller, Deep learning for time series classification: a review, Data mining and knowledge discovery 33 (4) (2019) 917–963.
  * [2] Z. Wang, J. Chen, S. C. Hoi, Deep learning for image super-resolution: A survey, IEEE transactions on pattern analysis and machine intelligence 43 (10) (2020) 3365–3387.
  * [3] Z. Bai, X.-L. Zhang, Speaker recognition based on deep learning: An overview, Neural Networks 140 (2021) 65–99.
  * [4] L. Böttcher, N. Antulov-Fantulin, T. Asikis, Ai pontryagin or how artificial neural networks learn to control dynamical systems, Nature communications 13 (1) (2022) 333.
  * [5] Y. Wang, X. Du, Z. Lu, Q. Duan, J. Wu, Improved lstm-based time-series anomaly detection in rail transit operation environments, IEEE Transactions on Industrial Informatics 18 (12) (2022) 9027–9036.
  * [6] B. Wei, L. Yang, N. Xie, Nonlinear grey bernoulli model with physics-preserving cusum operator, Expert Systems with Applications 229 (2023) 120466.
  * [7] W. Xie, C. Liu, W.-Z. Wu, A novel fractional grey system model with non-singular exponential kernel for forecasting enrollments, Expert Systems with Applications 219 (2023) 119652.
  * [8] S.-Y. Chen, G.-F. Qu, X.-H. Wang, H.-Z. Zhang, Traffic flow forecasting based on grey neural network model, in: Proceedings of the 2003 International Conference on Machine Learning and Cybernetics (IEEE Cat. No. 03EX693), Vol. 2, IEEE, 2003, pp. 1275–1278.
  * [9] F. Wu, R. Jing, X.-P. Zhang, F. Wang, Y. Bao, A combined method of improved grey bp neural network and meemd-arima for day-ahead wave energy forecast, IEEE Transactions on Sustainable Energy 12 (4) (2021) 2404–2412.
  * [10] D. Lei, K. Wu, L. Zhang, W. Li, Q. Liu, Neural ordinary differential grey model and its applications, Expert Systems with Applications 177 (2021) 114923.
  * [11] X. Ma, M. Xie, J. A. Suykens, A novel neural grey system model with bayesian regularization and its applications, Neurocomputing 456 (2021) 61–75.
  * [12] W. Xie, W.-Z. Wu, Z. Xu, C. Liu, K. Zhao, The fractional neural grey system model and its application, Applied Mathematical Modelling (2023).
  * [13] J. He, S. Mao, A. K. Ng, Neural computing for grey richards differential equation to forecast traffic parameters with various time granularity, Neurocomputing 549 (2023) 126394.
  * [14] M. Raissi, P. Perdikaris, G. E. Karniadakis, Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations, Journal of Computational physics 378 (2019) 686–707.
  * [15] L. Du, Z. Wang, Z. Lv, L. Wang, D. Han, Research on underwater acoustic field prediction method based on physics-informed neural network, Frontiers in Marine Science 10 (2023) 1302077.
  * [16] D. Li, B. Zhao, S. Lu, J. Wang, Physics-informed dynamic mode decomposition for short-term and long-term prediction of gas-solid flows, Chemical Engineering Science (2024) 119849.
  * [17] F. Iglesias-Suarez, P. Gentine, B. Solino-Fernandez, T. Beucler, M. Pritchard, J. Runge, V. Eyring, Causally-informed deep learning to improve climate models and projections, Journal of Geophysical Research: Atmospheres 129 (4) (2024) e2023JD039202.
  * [18] J. Vanterler, D. Sousa, E. Capelas, D. Oliveira, A new truncated m-fractional derivative type unifying some fractional derivative types with classical properties, Int. J. Anal. Appl 16 (1) (2018) 83–96.
  * [19] S. Liu, Y. Yang, N. Xie, J. Forrest, New progress of grey system theory in the new millennium, Grey Systems: Theory and Application 6 (1) (2016) 2–31.
  * [20] D. Bainov, P. Simeonov, Integral inequalities application, New York: Springer (1992).
  * [21] J. Wang, P. Du, H. Lu, W. Yang, T. Niu, An improved grey model optimized by multi-objective ant lion optimization algorithm for annual electricity consumption forecasting, Applied Soft Computing 72 (2018) 321–337.
  * [22] C. Zanchettin, T. B. Ludermir, L. M. Almeida, Hybrid training method for mlp: optimization of architecture and training, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 41 (4) (2011) 1097–1109.
  * [23] X. Ma, W. Wu, B. Zeng, Y. Wang, X. Wu, The conformable fractional grey system model, ISA transactions 96 (2020) 255–271.
  * [24] L. Wu, S. Liu, L. Yao, S. Yan, D. Liu, Grey system model with the fractional order accumulation, Communications in Nonlinear Science and Numerical Simulation 18 (7) (2013) 1775–1785.
  * [25] C. Yan, W. Lifeng, L. Lianyi, Z. Kai, Fractional hausdorff grey model and its properties, Chaos, Solitons & Fractals 138 (2020) 109915.
  * [26] M. Mao, E. C. Chirwa, Application of grey model gm (1, 1) to vehicle fatality risk estimation, Technological Forecasting and Social Change 73 (5) (2006) 588–605.
  * [27] N.-m. Xie, S.-f. Liu, Discrete grey forecasting model and its optimization, Applied mathematical modelling 33 (2) (2009) 1173–1186.


# Mashee at SemEval-2024 Task 8: The Impact of Samples Quality on the
Performance of In-Context Learning for Machine Text Classification

Areeg Fahad Rasheed
College of Information Engineering
Al-Nahrain University
Baghdad, Iraq
<EMAIL_ADDRESS>
M. Zarkoosh
Software Engineering
Baghdad, Iraq
<EMAIL_ADDRESS>

###### Abstract

Within few-shot learning, in-context learning (ICL) has become a potential
method for leveraging contextual information to improve model performance on
small amounts of data or in resource-constrained environments where training
models on large datasets is prohibitive. However, the quality of the selected
sample in a few shots severely limits the usefulness of ICL. The primary goal
of this paper is to enhance the performance of evaluation metrics for in-
context learning by selecting high-quality samples in few-shot learning
scenarios. We employ the chi-square test to identify high-quality samples and
compare the results with those obtained using low-quality samples. Our
findings demonstrate that utilizing high-quality samples leads to improved
performance with respect to all evaluated metrics.

##  1 Introduction

The advent of large language models (LLMs) like GPT-3.5 has brought about
transformative capabilities, seamlessly handling tasks like question
answering, essay writing, and problem-solving Aljanabi et al. (2023); Wu et
al. (2023); Rasheed et al. (2023a). However, this technological advancement
necessitates careful consideration of its associated challenges. Concerns
regarding the potential impact on creativity and ethical implications,
particularly concerning the generation of deepfakes Tang et al. (2023),
warrant careful attention RAYMOND (2023). Additionally, the limitations of
LLMs, including the possibility of producing erroneous information, require
rigorous evaluation and verification. The substantial energy consumption
required for training LLMs on massive datasets raises environmental concerns,
contributing to their carbon footprint. Moreover, plagiarism issues emerge as
users may misuse the generated content, either inadvertently or intentionally
Hadi et al. (2023).

Various models have been introduced in recent years designed to distinguish
text generated by humans from that created by machinesMitchell et al. (2023).
Examples include GPTZerogpt , AI Content Detectorcop , and AI Content Detector
by Writer wri  among others. Some of these models are trained on specific
datasets, while others are commercially available. Designing and implementing
LLMs for classification tasks requires substantial resources and computational
power, which are often only accessible to institutions and governments.
Therefore, various optimization models, such as LoRA Hu et al. (2021),
distillationHsieh et al. (2023), quantizationDettmers et al. (2022), and in-
context learning Liu et al. (2022), have been developed to reduce the resource
requirements for LLM implementation. This paper focuses on In Context Learning
(ICL) Liu et al. (2022), which utilizes the capabilities of other models to
enhance their ability to classify AI-generated text.

In Context Learning (ICL) is a Natural Language Processing (NLP) technique
utilized to enable Large Language Models (LLMs) to learn new tasks based on
minimal examples. This technique proves powerful in scenarios where training
models on extensive datasets is impractical or when there are constraints on
dataset availability for a specific task. ICL operates on the premise that
humans can often acquire new tasks through analogy or by observing a few
examples of task performance. It can be employed without any examples and is
referred to as zero-shot learning. Alternatively, if the input includes one
example, it is termed one-shot learning, and if it contains more than one, it
is known as few-shot learning. This paper focuses on the application of few-
shot learning within the context of ICLAhmed and Devanbu (2022); Kang et al.
(2023).

In this study, our focus lies exclusively on few-shot learning. We present a
methodology that leverages the chi-square statistic Rasheed et al. (2023b);
Lancaster and Seneta (2005) to select samples for few-shot learning and
evaluate its impact on the performance of a machine-generated text
classification model. We work on task A English language only Wang et al.
(2024).

##  2 Dataset

The dataset employed for Task A comprises two main components. The first part,
derived from human writing, was collected from diverse sources including
WikiBidia, WikiHow, Reddit, ArXiv, and PeerRead. The second part consists of a
machine-generated text produced by ChatGPT, Cohere, Dolly-v2, and
BLOOMzMuennighoff et al. (2023). For further details, please refer to the
associated paper Wang et al. (2023).

##  3 Chi-square

Chi-square is a statistical test used to assess the independence of two
categorical variables. It calculates the difference between observed and
expected frequencies of outcomes, and a larger chi-square value indicates a
stronger rejection of independence. In text analysis, chi-square can be used
to identify keywords that are more likely to occur in one category than
another, making it useful for feature selection and text classification. We
computed the chi-square values for each training sample and recorded the
sample index with the highest and lowest chi-square values for both human-
generated and machine-generated samples. Table I displays the index and
corresponding chi-square values for each of these instances. We will use
$X^{2}$ to refer to chi-square Lancaster and Seneta (2005).

Table 1: Indices and chi-square values for highest/lowest in human-generated and machine-generated text Name | Index # |  $X^{2}$ Value
---|---|---
Highest $X^{2}$ (Human) | 70873 | 1351.59
Lowest $X^{2}$ (Human) | 85726 | 1.21
Highest $X^{2}$ (Machine) | 2426 | 1154.27
Lowest $X^{2}$ (Machine) | 29111 | 0.8243

##  4 System overview

Figure 1: Proposed System Components

The system architecture is illustrated in Figure 1. The process starts with
feeding the entire training dataset to a chi-square computation, where the
chi-square value for each sample is calculated. Subsequently, the indices of
the samples with the highest and lowest chi-square values are selected for
both human-generated and machine-generated datasets using information from
Table I. Next, context learning is prepared. Initially, multiple templates
were tested, and the one presented in Figure 1 yielded the best results. This
template is then fed with two samples: the first being the machine-generated
sample with the highest chi-square value, and the second being the human-
generated sample with the highest chi-square value. Due to context window size
limitations, only the first 5000 characters of each sample are incorporated.
This is applied to training samples exceeding 5000 characters to ensure the
context learning size is not exceeded. Finally, the test sample is fed into
the context-learning process. The Flan-T5 model large version is used. The
results are then recorded and evaluated. The dev/test sample size was
truncated to 3000. We also evaluated the system using samples with the lowest
chi-square values and doing the same process.

Dataset | Chi Type | Recall | Precision | F1-Score | Accuracy
---|---|---|---|---|---
Dev set | Lowest | 46.92 | 46.90 | 46.84 | 46.92
Highest | 53.76 | 53.76 | 53.74 | 53.76
Test set | Lowest | 55.04 | 55.07 | 55.03 | 55.27
Highest | 58.68 | 58.81 | 58.81 | 55.99
Table 2: Experiments results

##  5 Findings and Analysis

We employed the Flan-T5 Large model for both the development and testing
datasets. We selected samples from both human-generated and machine-generated
sources, with each sample limited to 5000 characters to avoid exceeding the
token size limit. A total of four experiments were conducted. The first
experiment utilized samples with high chi-square values from the development
set. The second experiment focused on samples with the smallest chi-square
values from the development set. The third experiment involved samples with
high chi-square values from the test set. Finally, the fourth experiment
utilized samples with low chi-square values from the test set. Table II
presents all achieved results.

Based on the results presented in Table II, we can discuss several key points.

  * •

The results highlight the crucial role of sample quality in the performance of
in-context learning. By leveraging the chi-squared metric and prioritizing
samples with high values, we essentially provide the Flan-T5 model with
examples rich in diverse features. This choice enables the Flan-T5 model to
learn more effectively, drawing substantial insights from the samples.
Consequently, the model becomes more familiar with the provided data,
ultimately enhancing its performance. In contrast, selecting samples with
lower quality leads to less optimal performance. This can be noticed for both
the dev and test set. The main reason behind this is that words in the sample
with high chi-square values contain the most distinctive features. This is
because the chi-square test assigns high values to words that are frequent
within a particular class but appear less frequently in other
classes.Conversely, samples with lower chi-square values likely contain more
random words that appear with similar frequency across all classes. In chi-
square analysis, words that appear equally or approximately equally in each
class receive lower scores.

  * •

The classification of machine-generated text represents a novel frontier in
machine learning, and the availability of datasets for this task is currently
limited. The dataset used in this study was generated in 2023, marking it as a
recent development and underscoring the lack of established benchmarks. Models
that support in-context learning have not been trained extensively on such
tasks, resulting in lower accuracy when applied. While examples with high-
quality data can enhance model performance, it remain below the desired
threshold. Hence, it is advisable to train the model directly on the dataset
rather than relying on in-context learning.

  * •

We have utilized the Flan-T5 model; however, other models can be employed to
evaluate the performance of text classification machinery. We suggest
considering alternatives such as bard, Jurassic-1 Jumbo, and ChatGPT.

##  6 Conclusion

This work presents a system for classifying human-generated and machine-
generated text. The system leverages the combined strengths of in-context
learning and Chi-square analysis. Chi-square is employed to select high-
quality samples from the trainin dataset for few-shot learning in the in-
context learning. We implement Flan-T5 model large version for in-context
learning. Evaluation using accuracy, recall, precision, and F1-score
demonstrates that selecting high-quality samples improves system performance
for both dev and test. Furthermore, the results indicate that relying solely
on in-context learning for new tasks like machine-generated text detection
yields relatively low performance.

## References

  * (1) Ai content detector.  Accessed on .
  * (2) Ai content detector by writer.  Accessed on .
  * (3) Gptzero.  Accessed on .
  * Ahmed and Devanbu (2022) Toufique Ahmed and Premkumar Devanbu. 2022.  Few-shot training llms for project-specific code-summarization.  In _Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering_ , pages 1–5.
  * Aljanabi et al. (2023) Mohammad Aljanabi, Mohanad Ghazi, Ahmed Hussein Ali, Saad Abas Abed, et al. 2023.  Chatgpt: open possibilities.  _Iraqi Journal For Computer Science and Mathematics_ , 4(1):62–64.
  * Dettmers et al. (2022) Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022.  Llm. int8 (): 8-bit matrix multiplication for transformers at scale.  _arXiv preprint arXiv:2208.07339_.
  * Hadi et al. (2023) Muhammad Usman Hadi, R Qureshi, A Shah, M Irfan, A Zafar, MB Shaikh, N Akhtar, J Wu, and S Mirjalili. 2023.  A survey on large language models: Applications, challenges, limitations, and practical usage.  _TechRxiv_.
  * Hsieh et al. (2023) Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023.  Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes.  _arXiv preprint arXiv:2305.02301_.
  * Hu et al. (2021) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021.  Lora: Low-rank adaptation of large language models.
  * Kang et al. (2023) Sungmin Kang, Juyeon Yoon, and Shin Yoo. 2023.  Large language models are few-shot testers: Exploring llm-based general bug reproduction.  In _2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)_ , pages 2312–2323. IEEE.
  * Lancaster and Seneta (2005) Henry Oliver Lancaster and Eugene Seneta. 2005.  Chi-square distribution.  _Encyclopedia of biostatistics_ , 2.
  * Liu et al. (2022) Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. 2022.  Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning.  _Advances in Neural Information Processing Systems_ , 35:1950–1965.
  * Mitchell et al. (2023) Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. 2023.  Detectgpt: Zero-shot machine-generated text detection using probability curvature.  _arXiv preprint arXiv:2301.11305_.
  * Muennighoff et al. (2023) Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2023.  Crosslingual generalization through multitask finetuning.
  * Rasheed et al. (2023a) Areeg Fahad Rasheed, M Zarkoosh, Safa F Abbas, and Sana Sabah Al-Azzawi. 2023a.  Arabic offensive language classification: Leveraging transformer, lstm, and svm.  In _2023 IEEE International Conference on Machine Learning and Applied Network Technologies (ICMLANT)_ , pages 1–6. IEEE.
  * Rasheed et al. (2023b) Areeg Fahad Rasheed, M Zarkoosh, and Sana Sabah Al-Azzawi. 2023b.  The impact of feature selection on malware classification using chi-square and machine learning.  In _2023 9th International Conference on Computer and Communication Engineering (ICCCE)_ , pages 211–216. IEEE.
  * RAYMOND (2023) DANIEL RAYMOND. 2023.  Disadvantages of large language models.  Accessed on .
  * Tang et al. (2023) Ruixiang Tang, Yu-Neng Chuang, and Xia Hu. 2023.  The science of detecting llm-generated texts.  _arXiv preprint arXiv:2303.07205_.
  * Wang et al. (2023) Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Chenxi Whitehouse, Osama Mohammed Afzal, Tarek Mahmoud, Alham Fikri Aji, and Preslav Nakov. 2023.  M4: Multi-generator, multi-domain, and multi-lingual black-box machine-generated text detection.
  * Wang et al. (2024) Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Chenxi Whitehouse, Osama Mohammed Afzal, Tarek Mahmoud, Giovanni Puccetti, Thomas Arnold, Alham Fikri Aji, Nizar Habash, Iryna Gurevych, and Preslav Nakov. 2024.  SemEval-2024 task 8: Multigenerator, multidomain, and multilingual black-box machine-generated text detection.  In _Proceedings of the 18th International Workshop on Semantic Evaluation_ , SemEval 2024, Mexico City, Mexico.
  * Wu et al. (2023) Tianyu Wu, Shizhu He, Jingping Liu, Siqi Sun, Kang Liu, Qing-Long Han, and Yang Tang. 2023.  A brief overview of chatgpt: The history, status quo and potential future development.  _IEEE/CAA Journal of Automatica Sinica_ , 10(5):1122–1136.