11institutetext: University of Insubria, Varese, Italy 22institutetext:
University of Essex, Colchester, Essex

# Binary Classification using Pairs of Minimum Spanning Trees or N-ary Trees

Riccardo La Grassa 11 0000-0002-4355-0366 Ignazio Gallo 11 0000-0002-7076-8328
Alessandro Calefati 11 0000-0003-3860-4785 Dimitri Ognibene 22
0000-0002-9454-680X

###### Abstract

One-class classifiers are trained only with target class samples. Intuitively,
their conservative modeling of the class description may benefit classical
classification tasks where classes are difficult to separate due to
overlapping and data imbalance. In this work, three methods leveraging on the
combination of one-class classifiers based on non-parametric models, N-ary
Trees and Minimum Spanning Trees class descriptors (MST_CD) are proposed.

These methods deal with inconsistencies arising from combining multiple
classifiers and with spurious connections that MST-CD creates in multi-modal
class distributions. Experiments on several datasets show that the proposed
approach obtains comparable and, in some cases, state-of-the-art results.

###### Keywords:

One-class Classifiers Minimum Spanning Tree Instance-based approaches Non-
Parametric models

##  1 Introduction

With the rise of social platforms and internet, data produced by users has
grown exponentially enabling the use of data greedy machine learning
algorithms in several applications, still in many domains of practical
interests, data are still scarce and require more data efficient methods
especially for non-trivial classification tasks where classes are difficult to
separate due to overlapping and data imbalance. One-class classifiers are
trained with target class only samples under the strong assumption that data
from the other classes are not available or have low quality. Intuitively,
their conservative modeling of the data distribution may benefit classical
classification tasks if they were combined in a effective manner.

In this work, three methods are proposed which leverage on the combination of
one-class classifiers based on non-parametric models, K-Nearest Neighbour,
Trees and Minimum Spanning Trees class descriptors (MST-CD), to tackle binary
classification problems.

In the first model, we train classifiers using Minimum Spanning Tree Class
Descriptor (MST_CD) in the training step and then apply a new technique to
provide a more reliable prediction. The second model creates a more powerful
classifier based on MST_CD combining results according to an ensemble method.
The third model is very similar to the previous one but uses a tree starting
to the closest neighbour to the target pattern for each classifier and finally
it leverages on the ensemble technique.

In the next Section related works are shown, the proposed approach is
described in Section 4, Experiments are in Section 6 and Conclusions in 7.

##  2 Related work

In the feature selection field, many approaches have been proposed and used
with classifiers to obtain better accuracy [2, 7, 1, 12]. Krawczyk et al. [8]
proposes a generic model that improves the performance of many common
classifiers showing des standard and des-threshold methodologies. They are
based on a k-Nearest Neighbour (k-NN) technique that assigns a pattern to a
class on the basis of the class of its nearest $k$ neighbours. The approach
shows good results and authors compare it with classical models. However, they
do not consider the case in which no classifier is “activated” and, thus, when
it is not possible to obtain a prediction for a new instance $x$. Duin et al.
[14] proposes a simple approach to assign these refused objects to the class
with largest prior probability, but they do not describe a method in the
scenario where the two decision boundaries overlap. Our approach combines part
of the approach described in [8][14] using MSTs with other methodologies to
improve the accuracy and considering also the overlapping. Abpeykar et al. [1]
proposes a survey that sums up the performance of many classifiers on well-
known datasets from UCI repository. A milestone on one-class classifier comes
from Pekalska et al. [6] with their MST descriptor. The original idea was to
try to search a pattern from all training sets in order to create a MST that
represents the model on which will be done some geometrical operations with
the goal to generate a border for a specific class. Segui et al. [11] focuses
on the research of noise within a target class and removes it in order to have
better accuracy at testing time. They confirm that a graph-based one-class
classifier, like MST_CD obtains good results than other approaches, especially
dealing with small samples cardinalities and high data dimensionalities.
Quinlan [9] proposes a general method that allows predictions using both mixed
approach of instance-based and model-based learning. He proves that these
composite methods often produce better results in term of predictions than
using only a single methodology.

##  3 Minimum Spanning Tree class descriptor

As widely described in [6] a MST_CD is a non-parametric classifier able to
create a structure, seeing only data of the single class of interest. This
structure is based on Minimum Spanning Tree, basic elements of this classifier
are not only vertices but also edges of the graph, giving a richer
representation of the data. Considering edges of the graph as target class
objects, additional virtual target objects are generated. The classification
of a new object $x$ is based on the distance to the nearest vertex or edge.
The key of this classifier is to track a shape around the training set not
considering only all instances of the training but also edges of the graph, in
order to have more structured information. Therefore, in prediction phase, the
classifier considers two important elements:

  * •

Projection of point $x$ on a line defined by vertices ${x_{i},x_{j}}$

  * •

Minimum Euclidean distance between $(x,x_{i})$ and $(x,x_{j})$

The Projection of $x$ is defined as follow:

$p_{e_{i,j}}(x)=x_{i}+\frac{(x_{j}-x_{i})^{T}(x_{j}-x_{i})}{||x_{j}-x_{i}||^{2}}(x_{j}-x_{i})$

if $p_{e_{i,j}}(x)$ lies on the edge $e_{i,j}$= ($x_{i}$,$x_{j}$), we compute
$p_{e_{i,j}}(x)$ and the Euclidean distance between $x$ and $p_{e_{i,j}}(x)$,
more formally:

$0<=\frac{(x_{j}-x_{i})^{T}(x_{j}-x_{i})}{||x_{j}-x_{i}||^{2}}<=1$

then

$d(x|e_{i,j})=||x-p_{e_{i,j}}(x)||$

Otherwise we compute the Euclidean distance of $x$ and pairs ($x_{i}$,
$x_{j}$), precisely:

$d(x|e_{i,j})=min(||x-x_{j}||,||x-x_{i}||)$

Therefore, a new object $x$ is recognized by MST_CD if it lies in proximity of
the shape built in training phase, otherwise the object is considered as
outlier. The decision whether an object is recognized by classifier or not is
based on threshold of the shape created during the training phase, more
formally:

$d_{MST\\_CD}(x|X)<=\theta$

Authors set the threshold $\theta$ as the median of the distribution of the
edge weights $w_{ij}=||e_{ij}||$ in the given MST. Given
$\hat{e}=(||e_{1}||,||e_{2}||,...,||e_{n}||)$ as an ordered edge weights
values, they define $\theta$ as $\theta=||e_{[\alpha n]}||$, where
$\alpha\in[0,1]$. For instance, with $\alpha=0.5$, we assign the median value
of all edge weights into the MST.

##  4 Proposed Approach

The main objective of a one-class classifier is to recognize instances of a
selected class from a set of samples. All instances that are not classified by
this model will be considered as outliers (or alien class), while others will
be recognized as belonging to the same class of the training set. In this
context, we cannot say anything about the refused objects (outlier), but if we
have a one-class classifier for each label of the dataset we can not have
outliers because if a classifier refuses an object, it should be accepted by
the others classifiers. In this work, we use two one-class classifiers trained
on two different classes. We assume a discriminant function $f_{ab}$ on a
binary classification problem to classify a new object considering acceptation
and rejection from both our classifiers. Given $X$ a MST and $x$ a new object,
we define a function that assigns a label such that:

$f_{ab}(x,X)=\begin{cases}1&\text{if $d_{MST\\_CD_{0}}(x|X)<=\theta$ and
$d_{MST\\_CD_{1}}(x|X)>\theta_{1}$}\\\ 0&\text{if
$d_{MST\\_CD_{0}}(x|X)>\theta$ and
$d_{MST\\_CD_{1}}(x|X)<=\theta_{1}$}\end{cases}$ (1)

However, we may have two anomaly cases:

  1. 1.

both classifiers refuse the pattern to be predicted

  2. 2.

both classifiers accept the pattern to be predicted

In this specific case we can apply a simple technique to assign a pattern to
two possible classes depending on its distance to clusters. More formally,
given $x$ a target vector, $i$ and $j$ are all elements of the dataset
belonging to both classes, we define:

$d(x|i)=(|x-i_{0}|,|x-i_{1}|,...,|x-i_{n_{0}}|)$
$d^{\prime}(x|j)=(|x-j_{0}|,|x-j_{1}|,...,|x-j_{n_{1}}|)$

where $n_{0}$ and $n_{1}$ are cardinalities of first and second class
respectively.

Then we take the $k$ nearest elements for each class, such that:

$k<=min(n_{0},n_{1})$

We compute the vector difference as:

$Vector_{diff}=[(d_{1}-d^{\prime}_{1}),(d_{2}-d^{\prime}_{2}),...,(d_{k}-d^{\prime}_{k})]$

Finally, We assume a new function $f^{\prime}_{ab}$ to classify the object as:

$f^{\prime}_{ab}(x,Vector_{diff})=\begin{cases}1&\text{if
}|Vector_{diff}(k<=0)|>|Vector_{diff}(k>0)|\\\ 0&\text{otherwise}\end{cases}$
(2)

###  4.1 First approach

Our first approach combines two one-class classifiers based on Minimum
Spanning Tree class descriptor and solve both above mentioned issues. When one
of the two issues appear, e.g. both classifiers accept (classifiers overlap)
or reject (uncovered) the input sample, an approach similar to K-NN majority
vote is applied. Using the already computed euclidean distances between the
sample and the elements of the two MSTs, the K elements of each MST
($MST_{1}$,$MST_{2}$) nearer to the sample are selected to check which of the
two MSTs is consistently closer to the sample. This is done by internally
sorting the two sets of K elements in increasing distance order from the
sample then subtracting the 2 corresponding K-ary vectors ($D_{1}$, $D_{2}$)
of distances and finally counting how many positive elements are in the
resulting vector $R=D_{1}-D_{2}$. If $R$ contains more positive than negative
elements the sample is associated to $MST_{2}$ as its elements are closer to
the sample. This method allows to integrate the generalization capabilities of
the MST with the robustness offered by the K-NN vote strategy to deal with
binary classification strategies. See the pseudo-code in Algorithm 1.

Figure 1: MST_CD model: In this figure, we show the problem about the
competence area into two different classifiers in a toy scenario. In this case
the object $x$ will be predicted as blue class instead of green class. This
happens because the orthogonal projection of $x$ is nearer to an edge that
contains an outlier and the object is located into a non-competence area of
the blue-classifier making a wrong prediction. Algorithm 1 MST CD OVA

1:for $v\in\mathcal{G}_{0}$ do

2: all euclidean distances $\leftarrow||x-v||$

3:end for

4:NodeX = Take min(all euclidean distances)

5:EdgeNodeX $\leftarrow$ Search inc/out edge nodeX and return $(u,v)$

6:for $u,v\in\mathcal{E}(NodeX)$ do

7: if $0<=\frac{(x_{j}-x_{i})^{T}*(x-x_{i})}{||x_{j}-x_{i}||^{2}}<=1$ then

8:
$P_{e_{{}_{i_{j}}}}(x)=x_{i}+\frac{(x_{j}-x_{i})^{T}*(x-x_{i})}{||x_{j}-x_{i}||^{2}}*(x_{j}-x_{i})$

9: $d(x|e_{{}_{i_{j}}})\leftarrow||x-P_{e_{{}_{i_{j}}}}(x)||$

10: else

11: $d(x|e_{{}_{i_{j}}})\leftarrow
min\big{\\{}||x-x_{i}||,||x-x_{j}||\big{\\}}$

12: end if

13:end for

14:Repeat line 1-13 for graph $G_{1}$

15:min dist0 = $min(d(x|e_{ij}))$

16:min dist1 = $min(d1(x|e_{ij}))$

17:1 $\leftarrow$ if $d_{MST\\_CD_{0}}(x|X)<=\theta$ and
$d_{MST\\_CD_{1}}(x|X)>\theta_{1}$

18:0 $\leftarrow$ if $d_{MST\\_CD_{0}}(x|X)>\theta$ and
$d_{MST\\_CD_{1}}(x|X)<=\theta_{1}$

19:if min dist0 $<=\theta$ and min dist1 $<=\theta_{1}$ then

20: knn weight1=order($d_{1}(x|u)$) and take $k_{1}$-elements

21: knn weight0=order($d_{0}(x|v)$) and take $k_{1}$-elements

22: euclidean distance vectors = (knn weight1 - knn weight0)

23: positive $=$ Count $n_{i}>0$ in euclidean distance vectors

24: negative $=$ Count $n_{i}<0$ in euclidean distance vectors

25: if negative $>=$ positive then

26: prediction $\leftarrow 1$

27: else

28: prediction $\leftarrow 0$

29: end if

30:end if

31:if min dist0 $>\theta$ and min dist1 $>\theta_{1}$ then

32: The approach is equal to lines [20-29]

33:end if

###  4.2 Second approach

In a second approach, we mix K-NN and MST in the reverse order. Using a K-NN
like approach, we initially select the K training samples from each class
closest to the X sample to classify. This results in two sets $S_{1}$ and
$S_{2}$, we then create two K-elements MST for each of this sets
($kMST_{1}$,$kMST_{1}$). After this we classify X as we did for approach 1.
This second approach has two advantages over the previous. One, is that it
does not need to perform the expensive creation of the two large MST covering
the whole dataset (quadratic in the number of samples) but only deals with a
K-elements MSTs. Second, it avoids MST spurious edges between elements of
distant and unrelated modes of a class distribution. See the pseudo-code in
Algorithm 2 and Fig.2.

Figure 2: MST_CD_GP model: A different approach on the same toy example
considering a combination built on two small MST starting from nodes with a
minimum distance from target pattern. Algorithm 2 MST CD with Gamma parameters

1:function create small MST(g0 weight sorted, g1 weight sorted)

2: $smallG_{0}$ = first gamma index sorted values in g0 weight sorted

3: edges couple $\leftarrow$ all combinations nodes small g0

4: for $u,v\in\mathcal{e}dgescouple$ do

5: $small\ G_{0}\leftarrow(node\ u,node\ v,weight=(u,v))$

6: end for

7: $small\ MST_{0}=ComputeMST(small\ G_{0})$

8: Same approach for $MST_{1}$

9: $e(small\ MST0)=(||e_{0}||,||e_{1}||,..||e_{n}||)$

10: $e(small\ MST1)=(||e_{0}||,||e_{1}||,..||e_{n}||)$

11: $\theta_{0}=||e_{(\alpha n)}||$

12: $\theta_{1}=||e_{(\alpha n)}||$

13:return small $MST_{0}$, $smallMST_{1}$

14:end function

###  4.3 Third approach

The third is similar to the second one but instead of selecting the $K$
nearest elements to the sample $X$ to classify, it selects only the nearest
element $E$ to the sample and then selects the K-1 elements of the training
set nearest to $E$. It then creates a tree having $E$ as root and it uses it
as a classifier like in previous cases. This further extends the robustness to
outliers and spurious connections between far nodes. See the pseudo-code in
Algorithm 3 and Fig.3.

Figure 3: N-ary model: A different approach on the same toy example. Using a
N-ary model makes less complex the prediction phase ignoring outliers from the
structure. Algorithm 3 N-ary model

1:function create_n_ary($subG_{0}\ weight\ sorted,subG_{1}\ weight\ sorted,\
gamma\ index$)

2: $n\\_ary_{0}=first\ gamma\ index\ sorted\ values\ in\ g0\ weight\ sorted$

3: $n\\_ary_{1}=first\ gamma\ index\ sorted\ values\ in\ g1\ weight\ sorted$

4: for $u,v\in\mathcal{s}ubG\\_0weightedsorted$ do

5: $n\\_ary_{0}\leftarrow(nodeu,nodev,weight=(u,v))$

6: end for

7: for $u,v\in\mathcal{s}ubG\\_1weightedsorted$ do

8: $n\\_ary_{1}\leftarrow(nodeu,nodev,weight=(u,v))$

9: end for

10: return $n\\_ary_{0},n\\_ary_{1}$

11:end function

##  5 Datasets

We use five low-dimensional datasets (Hill, Sonar, Australian, Mofn, Pima) and
three high-dimensional datasets (Arcene, Gisette, Madelon), all taken from the
UCI repository [3]. All of them have two classes. Table 1 shows the details of
all datasets. To evaluate our approach in a more robust way, we selected
datasets with a huge variability in term of number of features and number of
instances. For all the datasets we use 5-fold Cross Validation. We replace the
missing values in the datasets with the average value for the missing
features.

Table 1: Number of features, classes, instances and positive-negative samples for all the datasets used in our experiment. Datasets | Features | Classes | Instances | pos-neg
---|---|---|---|---
Arcene | 10000 | 2 | 100 | 44-56
Gisette | 5000 | 2 | 6000 | 3000-3000
Madelon | 500 | 2 | 2000 | 1000-1000
Hill | 101 | 2 | 606 | 305-301
Sonar | 60 | 2 | 208 | 97-111
Australian | 14 | 2 | 690 | 307-383
Mofn | 10 | 2 | 1324 | 292-1032
Pima | 8 | 2 | 688 | 305-301

##  6 Experiments

In this section we report two groups of experiments,

  * •

to study the effect on the parameters in our models,

  * •

to compare the proposed models with the results available in the literature on
some well known benchmarks.

###  6.1 Parameters evaluation

For each classifier, we extract some common metrics to evaluate the
performance, such as Sensitivity, Precision and F1 Score. Starting from these
values we create confusion matrix and obtain the final accuracy considering
True/False positive and True/False negative samples.

Table 2: These are the results of both classifiers using our MST_CD model
about Sensitivity, Precision, F1 Score and Total accuracy on Sonar dataset.
Last line contains the average for each column. MST_CD
---
$Fold$ | $Sensitivity^{0}$ | $Sensitivity^{1}$ | $Precision^{0}$ | $Precision^{1}$ | $F1^{0}$ | $F1^{1}$ | $Accuracy$
1 | 0,867 | 0,900 | 0,867 | 0,900 | 0,867 | 0,900 | 0,886
2 | 0,833 | 0,944 | 0,938 | 0,850 | 0,882 | 0,895 | 0,889
3 | 0,800 | 1,000 | 1,000 | 0,833 | 0,889 | 0,909 | 0,900
4 | 0,688 | 0,880 | 0,786 | 0,815 | 0,733 | 0,846 | 0,805
5 | 0,714 | 0,850 | 0,769 | 0,810 | 0,741 | 0,829 | 0,794
Average | 0,780 | 0,915 | 0,872 | 0,842 | 0,822 | 0,876 | 0,855
Table 3: Details of our third model N-ary considering the same measures than
above table. We show an accuracy of 87.3% with gamma=20 and gamma=10 in N-ary
model against 85.5% in our first model. Results of each lines are extracted
using an average on k=5 fold-validation. N-ary
---
$Gamma$ | $Sensitivity^{0}$ | $Sensitivity^{1}$ | $Precision^{0}$ | $Precision^{1}$ | $F1^{0}$ | $F1^{1}$ | $Accuracy$
30 | 0,791 | 0,915 | 0,873 | 0,850 | 0,829 | 0,881 | 0,860
20 | 0,793 | 0,915 | 0,873 | 0,850 | 0,830 | 0,881 | 0,861
10 | 0,820 | 0,915 | 0,879 | 0,869 | 0,847 | 0,890 | 0,873
8 | 0,830 | 0,906 | 0,869 | 0,876 | 0,847 | 0,890 | 0,873
6 | 0,816 | 0,880 | 0,849 | 0,854 | 0,830 | 0,865 | 0,851
4 | 0,808 | 0,882 | 0,852 | 0,848 | 0,826 | 0,863 | 0,848
3 | 0,787 | 0,861 | 0,830 | 0,832 | 0,804 | 0,844 | 0,829
Table 4: Parameters set Parameters | Arcene | Madelon | Gisette | Mofn | Australian | Pima | Sonar | Hills
---|---|---|---|---|---|---|---|---
Threshold | 0.25 | 0.25 | 0.2 | 0.05 | 0.05 | 0.05 | 0.1 | 0.9
Neighbours | 4 | 12 | 4 | 2 | 6 | 6 | 2 | 2
Figure 4: Best Gamma parameter search for method using the Sonar dataset. We
use the average accuracy to select the best parameter.

###  6.2 Comparison

We evaluated the proposed models on the well known benchmark datasets
described in Section 5. In particular, we compared our methods using the
results published in [7, 1] and shown the comparison in Tables 5, 6 and 7. In
our work, we have two one-class classifiers and the rate of True/Negative
samples has been computed considering two different models. For instance if an
object has been predicted correctly True positive by a classifier the others
must be refuse and label it as True negative, otherwise, we consider all the
possible combinations ($2^{2}$) in our case (two classifiers). The final
accuracy is obtained from average of Cross-Validation. In our experiments, we
set $\gamma$ parameter into the second and third model within the range [2,30]
nodes and take the maximum accuracy obtained from Cross-Validation methods
considering always the same dimension training/test (80%-20%) see Table 3.
Threshold measures and k-neighbours are set as we show into Table 4 on each
specific dataset. In Table 5 we compare results of our models with many
references used to examine the performance of classifiers (Moradi & Rostami,
2015). Our models do not use feature selection techniques, therefore for each
instance we exploit all available features. Comparing our models with the
ensemble methods known as AdaboostM1 [4], Bagging [15], Dagging [15],
LogitBoost [5], MODLEM [13], Decorate (Melville & Mooney, 2003), Grading [10],
MultiBoostAB [16] and StackingC (Seewald, 2002), our models overcome the
accuracy of other ensembles classifier (except for Arcene dataset where
Bagging, LogitBoost and Modlem have higher accuracy than our models).

Table 5: 5-fold Cross-Validation accuracy results on $6$ datasets. Results
obtained with our three models (MST_CD, MST_CD_GP, N-ary) are compared with
results published in [7] on same datasets.

| Krakovna et al. [7] | Our models
---|---|---
Dataset | Bart | c5.0 | Cart | Lasso | LR | NB | RF | SBFC | SVM | TAN | CD | CD_GP | N-ary
Arcene | 71.6 | 66 | 63 | 65.6 | 52 | 69 | 71.8 | 72.2 | 72 | - | 79.6 | 77.7 | 80.3
Madelon | 76 | 75.8 | 78.2 | 60.7 | 60 | 59.8 | 67.1 | 63.4 | 62 | 54.2 | 75 | 75.2 | 75.3
Gisette | 97.7 | 94.8 | 90.8 | 97.2 | 88.1 | 90.3 | 97 | 95.2 | 96.9 | - | 96.8 | - | -
Mofn | 100 | 84.8 | 83.9 | 100 | 100 | 86.4 | 92.4 | 86.2 | 94.6 | 92.1 | 100 | 100 | 100
Australian | 86.9 | 86.7 | 84.2 | 85.6 | 86.8 | 85.7 | 87.8 | 86.9 | 86 | 86.8 | 69.0 | 70.6 | 70.3
Pima | 78.2 | 76.8 | 75.7 | 78 | 78.3 | 78 | 77.8 | 78.9 | 78.1 | 78.9 | 70.1 | 71.7 | 74.1

Table 6: 5-fold Cross-Validation accuracy results on five different datasets.
Results obtained with our three models (MST_CD, MST_CD_GP, N-ary) are compared
with the results of different ensemble methods published in [1].

| Abpeykar et al. [1] | Our models
---|---|---
Dataset | AdaB. | Bagg. | Dagg. | LogitB. | Mod. | Decor. | Grad. | Mt.B | Stack.C | CD | CD_GP | N-ary
Arcene | 79.5 | 82.5 | 74.5 | 85.5 | 86.0 | - | 56.0 | 80.0 | 56.0 | 79.6 | 77.7 | 80.3
Madelon | 63.4 | 75.0 | 57.2 | 63.0 | 52.5 | 73.3 | 50.1 | 61.7 | 50.1 | 75 | 75.2 | 75.3
Sonar | 71.6 | 76.9 | 69.7 | 79.3 | 70.6 | 84.1 | 53.3 | 74.5 | 53.3 | 85.4 | 85.2 | 88
Hills | 50.4 | 50.2 | 50.4 | 50.4 | - | - | 50.4 | 50.4 | 50.4 | 58.1 | 57.9 | 61.1
Gisette | 88.9 | 75.0 | 82.2 | 89.4 | - | 82.2 | 48.1 | 82.7 | 48.1 | 96.8 | - | -

Table 7: 5-fold Cross-Validation accuracy results on five datasets. The results obtained with our three models are compared with the results of different machine learning methods published in [1]. | Abpeykar et al. [1] | Our models
---|---|---
Datasets | Dt | k-NN | NB | RF | SVM | MST_CD | MST_CD_GP | N-ary
Arcene | 67.2 | 66.7 | 69.7 | 71.4 | 71.4 | 79.6 | 77.7 | 80.3
Madelon | 82.7 | 74.8 | 79.3 | 77.9 | 78.4 | 75 | 75.2 | 75.3
Sonar | 79.5 | 80.4 | 77.6 | 81.4 | 82.3 | 85.4 | 85.2 | 88.0

##  7 Conclusion

The presented results show that the solutions proposed are competitive both
with ensemble and classical classifiers. The second and third method that
aimed at increasing the robustness to outliers and spurious MST connections
were proved to consistently ameliorate accuracy. Also the latter methods avoid
the initial computation of expensive large MST while requiring the computation
of small MST at runtime and provide higher accuracy. This makes the latter
methods more convenient with large datasets for which the complete MST
computation may be too expensive. We are also confident that these methods can
be highly optimized using caching methods to further improve online
computation performance. Future work will tackle the issue of feature scaling
and selection as well as the possibility to combine the approach with ensemble
methods.

## References

  * [1] Abpeykar S., G.M., H., Z.: Ensemble decision forest of rbf networks via hybrid feature clustering approach for high-dimensional data classification. Computational Statistics and Data Analysis (2019)
  * [2] Budak, H., Taşabat, S.: A modified t-score for feature selection. Anadolu Üniversitesi Bilim Ve Teknoloji Dergisi A-Uygulamalı Bilimler ve Mühendislik (2016)
  * [3] Dua, D., Graff, C.: UCI machine learning repository (2019), http://archive.ics.uci.edu/ml
  * [4] Eibl, G., Pfeiffer, K.P.: How to make adaboost. m1 work for weak base classifiers by changing only one line of the code. In: European Conference on Machine Learning. pp. 72–83. Springer (2002)
  * [5] Frank, E., Holmes, G., Kirkby, R., Hall, M.: Racing committees for large datasets. In: International Conference on Discovery Science. pp. 153–164. Springer (2002)
  * [6] Juszczak, P., Tax, D.M., Pe, E., Duin, R.P., et al.: Minimum spanning tree based one-class classifier. Neurocomputing 72(7-9), 1859–1869 (2009)
  * [7] Krakovna, V., Du, J., Liu, J.S.: Interpretable selection and visualization of features and interactions using bayesian forests. Statistics and Its Interface 11, 503–513 (2018)
  * [8] Krawczyk, B., Galar, M., Woźniak, M., Bustince, H., Herrera, F.: Dynamic ensemble selection for multi-class classification with one-class classifiers. Pattern Recognition 83, 34–51 (2018)
  * [9] Quinlan, J.R.: Combining instance-based and model-based learning. In: Proceedings of the Tenth International Conference on Machine Learning. pp. 236–243 (1993)
  * [10] Seewald, A.K., Fürnkranz, J.: An evaluation of grading classifiers. In: International symposium on intelligent data analysis. pp. 115–124. Springer (2001)
  * [11] Segui, S., Igual, L., Vitrià, J.: Weighted bagging for graph based one-class classifiers pp. 1–10 (2010)
  * [12] Singh, B., Vyas, O.: Maximum spanning tree based redundancy elimination for feature selection of high dimensional data. International Arab Journal of Information Technology 15, 831–841 (09 2018)
  * [13] Stefanowski, J.: The rough set based rule induction technique for classification problems. In: In Proceedings of 6th European Conference on Intelligent Techniques and Soft Computing EUFIT. vol. 98 (1998)
  * [14] Tax, D.M., Duin, R.P.: Using two-class classifiers for multiclass classification. In: Object recognition supported by user interaction for service robots. vol. 2, pp. 124–127. IEEE (2002)
  * [15] Ting, K.M., Witten, I.H.: Stacking bagged and dagged models (1997)
  * [16] Webb, G.I.: Multiboosting: A technique for combining boosting and wagging. Machine learning 40(2), 159–196 (2000)


# On solutions of matrix-valued convolution equations, anisotropic fractional
derivatives and their applications in linear and non-linear anisotropic
viscoelasticity111An new version of [1] with improved notation and Theoorem
3.2.

Andrzej Hanyga
ul. Bitwy Warszawskiej 14/52
02-366 Warszawa, Poland

###### Abstract

A relation between matrix-valued complete Bernstein functions and matrix-
valued Stieltjes functions is applied to prove that the solutions of matricial
convolution equations with extended LICM kernels belong to special classes of
functions. In particular the cases of the solutions of the viscoelastic
duality relation and the solutions of the matricial Sonine equation are
discussed, with applications in anisotropic linear viscoelasticity and a
generalization of fractional calculus.

In the first case it is in particular shown that duality of completely
monotone relaxation functions and Bernstein creep functions in general
requires inclusion in the relaxation function of a Newtonian viscosity term in
addition to the memory effects represented by the completely monotone kernel.

We define anisotropic generalized fractional derivatives (GFD) by replacing
the kernel $t^{-\alpha}/\Gamma(1-\alpha)$ of the Caputo derivatives with
completely monotone matrix-valued kernels which are weakly singular at 0.

Notation.

  1. 1.

$\mathrm{D}=\mathrm{d}/\mathrm{d}t$

  2. 2.

$\mathbb{R}$ \- the set of real numbers

  3. 3.

$\mathbb{N}$ \- the set of non-negative integers

  4. 4.

$[a,b[$ \- the set of $x$ such that $a\leqslant x<b$

  5. 5.

$\mathcal{S}$ \- the set of symmetric matrices of rank N

  6. 6.

$\mathcal{S}_{+}$ \- the set of positive semi-definite elements of
$\mathcal{S}$

Keywords: viscoelasticity, anisotropic generalized fractional derivatives,
completely monotone, Bernstein, complete Bernstein, Stieltjes, Sonine
equation.

##  1 Introduction

A recent idea [4] of simplifying the proof of a viscoelastic duality relation
between the tensor-valued relaxation modulus and the tensor-valued creep
function [3], based on a relation between matrix-valued complete Bernstein
functions and matrix-valued Stieltjes functions [5] has led me to consider a
more general application of this method to convolutions of symmetric matrix-
valued functions.

In this reference I have decided to base the analysis of the viscoelastic
duality relation on the analysis of the solutions of the matrix-valued
solutions of the Sonine equation [9].

In the case of the Sonine equation it is assumed here that one of the
functions is locally integrable near zero and completely monotone (LICM). The
Sonine equation was examined in much detail in [8], but the authors did not
assume that one of the functions was LICM. They constructed the inverse
operator for the convolution equation $k(t)\ast x(t)=f(t)$. The inverse
operator however involves the solution $l(t)$ of the Sonine equation $k(t)\ast
l(t)=1$. Existence of such a function is the subject of our investigation and
we prove it for the kernel $k$ in the extended LICM class, defined below. The
last-mentioned problem is also studied in [6] for real-valued functions but we
consider more general symmetric matrix-valued functions.

I reexamine Kochubei’s suggestion [6] that the solutions of the Sonine
equation could be used to construct a generalization of the concepts of a
derivative and integration operators along the lines of fractional calculus. I
however allow for matrix-valued LICM kernels which introduce 3D modeling of
anisotropic effects in the context of non-linear relaxation equations.

I also show that in order to satisfy a Sonine equation the LICM function
$k(t)$ must be singular. If $k(t)$ is a singular LICM function, then it
satisfies the Sonine equation with an associated function $l(t)$, which is
also a singular LICM function.

The results presented here are relevant for anisotropic linear and non-linear
viscoelasticity. They also open new methods for dealing with problems
involving memory effects.

##  2 Convolutions equations for matrix-valued functions.

The convolution of two square matrix-valued functions $\mathbf{F}$ and
$\mathbf{G}$ of the same rank $N$ is defined by the equation

$(\mathbf{F}\ast\mathbf{G})(t):=\int_{0}^{t}\mathbf{F}(t-s)\,\mathbf{G}(s)\,\mathrm{d}s$
(1)

The Laplace transform $\tilde{\mathbf{F}}(p)$ of a matrix-valued function
$\mathbf{F}(t)$ is defined as usual by the formula

$\tilde{\mathbf{F}}(p):=\int_{0}^{\infty}\mathrm{e}^{-pt}\,\mathbf{F}(t)\,\mathrm{d}t$

for every $p\in\mathbb{C}$ such that the integral exists.

It is easy to check the identity

$\int_{0}^{\infty}\mathrm{e}^{-pt}\,\mathbf{F}(t)\ast\mathbf{G}(t)\,\mathrm{d}t=\tilde{\mathbf{F}}(p)\,\tilde{\mathbf{G}}(p),\;p>0$
(2)

for arbitrary matrix-valued functions defined on $[0,\infty[$ provided both
Laplace transforms on the right-hand side exist.

We shall equip the convolution algebra with a unit element $\mathrm{U}$:

$\mathrm{U}\,f=f$ (3)

(see Appendix A for details).

The unit operator $U$ is a convolution with a Borel measure $\upsilon$ on
$[0,\infty[$ defined by the formula
$\int_{[0,\infty]}f(r)\,\upsilon(\mathrm{d}r)=f(0)$ for every continuous
function $f$ on $[0,\infty[$. Indeed,

$(Uf)(r)=\int_{[0,\infty[}f(r-s)\,\,\upsilon(\mathrm{d}s)=f(r).$ (4)

The unit operator $U$ is a convolution with a Borel measure $\upsilon$ on
$[0,\infty[$ defined by the formula
$\int_{[0,\infty]}f(r)\,\upsilon(\mathrm{d}r)=f(0)$ for every continuous
function $f$ on $[0,\infty[$. Indeed,

$(Uf)(r)=\int_{[0,\infty[}f(r-s)\,\,\upsilon(\mathrm{d}s)=f(r).$

Extending identity (2) to (3) we have
$\tilde{\upsilon}(p)\tilde{f}(p)=\tilde{f}(p)$, whence

$\tilde{\upsilon}(p)=1,\;p\geqslant 0$ (5)

Consider the general convolutional equation

$\mathbf{A}_{1}\,\mathbf{X}(t)+\mathbf{F}(t)\ast\mathbf{X}(t)=\mathbf{R}(t)$
(6)

where $\mathbf{F}(t)$ and $\mathbf{R}(t)$ are two square matrix-valued
functions defined for $t\in\,]0,\infty[$ in a class to be specified,
$\mathbf{A}_{1}$ is a positive semi-definite symmetric matrix (possibly zero),
while $\mathbf{X}(t)$ is a square matrix-valued function defined by equation
(6). We shall examine the properties of the function $\mathbf{X}(t)$. The
ranks of the matrices are equal and will be denoted by $N$. Let $\mathcal{S}$
denote the space of symmetric real square matrices of a fixed rank $N$.

If $\mathbf{A}_{1}=0$ and $\mathbf{R}(t)$ is unit matrix for $t\geqslant 0$,
then equation (6) is a generalization of the Sonine equation [8] for matrix-
valued functions.

###### Definition 2.1

A matrix-valued function $\mathbf{F}:\,]0,\infty[\rightarrow\mathcal{S}$ is
said to be completely monotone (CM) if it is infinitely differentiable and for
every vector $\mathbf{v}\in\mathbb{R}^{N}$ the following inequalities are
satisfied

$\forall t>0\;\forall
n\in\mathbb{N}\;\;(-1)^{n}\,\mathrm{D}^{n}\mathbf{v}^{\mathsf{T}}\,\mathbf{F}(t)\,\mathbf{v}\geqslant
0$ (7)

The above definition allows for a singularity at 0.

###### Definition 2.2

A matrix-valued function $\mathbf{F}:\;]0,\infty[\rightarrow\mathcal{S}$ is
said to be locally integrable completely monotone (LICM) if it is CM and
integrable over $]0,1]$.

###### Theorem 2.1

[3]
Every matrix-valued LICM function $\mathbf{F}$ can be expressed in the form

$\mathbf{F}(t)=\int_{[0,\infty[}\mathrm{e}^{-rt}\,\mathbf{H}(r)\,\mu(\mathrm{d}r),\;t>0$
(8)

where $\mu$ is a Borel measure on $[0,\infty[$ satisfying the inequality

$\int_{[0,\infty[}(1+r)^{-1}\,\mu(\mathrm{d}r)<\infty$ (9)

and $\mathbf{H}(r)$ is a matrix-valued function on $[0,\infty[$ satisfying the
bound $|\mathbf{H}(r)|\leqslant 1$ except perhaps on a subset of $[0,\infty[$
of $\mu$-measure zero.

###### Proposition 2.3

If the matrix-valued function $\mathbf{F}$ is LICM, then the function
$\tilde{\mathbf{F}}(p)$ tends to 0 for $p\rightarrow\infty$.

Note that this result also implies that every LICM function $\mathbf{F}$
satisfies the identity

$\lim_{t\rightarrow 0}\int_{0}^{t}\mathbf{F}(s)\,\mathrm{d}s=0.$

Proof.
For $p>1$

$|\tilde{\mathbf{F}}(p)|=\int_{[0,\infty[}(p+r)^{-1}\,\mu(\mathrm{d}r)<\int_{[0,\infty[}(1+r)^{-1}\,\mu(\mathrm{d}r)<\infty,$

hence the thesis follows from the Lebesgue Dominated Convergence Theorem.

$\Box$

###### Definition 2.4

The set $\mathcal{E}$ consists of matrix-valued Borel measures of the form

$\Phi=\upsilon\,\mathbf{A}+\mathbf{F}\,\mathrm{d}t$

where $\mathbf{A}$ is a positive semi-definite symmetric matrix, $\mathbf{F}$
is an LICM $\mathcal{S}$-valued function and $\mathrm{d}t$ is the Lebesgue
measure on $[0,\infty[$.

The set $\mathcal{E}$ appears in a natural way in the context of solutions of
convolution equations. It also allows for a Newtonian viscosity component in
viscoelastic relaxation functions (Section 5).

###### Definition 2.5

A function $\mathbf{B}:\;]0,\infty[\rightarrow\mathcal{S}$ is said to be a
Bernstein function (BF) if $\mathbf{B}(t)$ is positive semi-definite and
differentiable for $t>0$ and its derivative is CM.

We denote by $\mathcal{B}$ is the set of all the Bernstein functions.

If $\mathbf{B}$ is an $\mathcal{S}$-valued Bernstein function, then for every
$\mathbf{v}\in\mathbb{R}^{N}$ the function
$t\rightarrow\mathbf{v}^{\mathsf{T}}\,\mathbf{B}(t)\,\mathbf{v}$, $\;t>0$, is
non-decreasing and continuous, hence it has a finite limit at $t=0$. Hence the
limit $\lim_{t\rightarrow 0}\mathbf{B}(t)$ exists. We shall therefore consider
Bernstein functions as defined on $[0,\infty[$ with
$\mathbf{B}(0)=\lim_{t\rightarrow 0}\mathbf{B}(t)$.

It follows that the derivative $\mathbf{B}^{\prime}$ of $\mathbf{B}$ on
$]0,\infty[$ is a completely monotone function. It is also locally integrable
because

$\int_{0}^{1}\mathbf{B}^{\prime}(t)\,\mathrm{d}t=\mathbf{B}(1)-\mathbf{B}(0),\;\mathbf{B}^{\prime}(t)\geqslant
0,$

hence it is LICM.

A general Bernstein function is obtained by integrating the measure
$\upsilon\,\mathbf{B}_{0}+\mathbf{F}\,\mathrm{d}t$ over intervals $[0,t]$ for
a symmetric and positive semi-definite matrix $\mathbf{B}_{0}$ and an LICM
matrix-valued function $\mathbf{F}$ .

An $\mathcal{S}$-valued LICM function $\mathbf{F}$ is locally integrable and
non-increasing, hence its Laplace transform $\tilde{\mathbf{F}}(p)$ is defined
for all $p>0$.

An $\mathcal{S}$-valued Bernstein function $\mathbf{B}$ also has a Laplace
transform defined for $p>0$ because

$\int_{0}^{\infty}\mathrm{e}^{-pt}\mathbf{B}(t)\,\mathrm{d}t=\int_{0}^{\infty}\mathrm{e}^{-pt}\,\int_{0}^{t}\mathbf{F}(s)\,\mathrm{d}s\,\mathrm{d}t=\frac{1}{p}\tilde{\mathbf{F}}(p),$

where $\mathbf{F}$ is the LICM derivative of $\mathbf{B}$.

Let $\mathcal{S}_{+}$ denote the set of positive semi-definite symmetric
$N\times N$ matrices.

###### Proposition 2.6

If $\mathbf{F}$ is a symmetric matrix-valued LICM function and for each vector
$\mathbf{v}\in\mathbb{R}^{N}$ the function
$\mathbf{v}^{\mathsf{T}}\,\mathbf{F}(t)\,\mathbf{v},t>0$, is not identically
zero, then the matrix $\tilde{\mathbf{F}}(p)$ is invertible for every $p>0$.

Proof.
For each non-zero $\mathbf{v}\in\mathbb{R}^{N}$ there is a real
$t_{1}(\mathbf{v})>0$ such that
$\mathbf{v}^{\mathsf{T}}\,\mathbf{F}(t_{1}(\mathbf{v}))\,\mathbf{v}>0$. The
function $\mathbf{v}^{\mathsf{T}}\,\mathbf{F}(t)\,\mathbf{v}$ is continuous,
hence it is positive on some interval $\mathcal{I}\subset]0,\infty[$, while it
is non-negative on $]0,\infty[$. Hence for every non-zero
$\mathbf{v}\in\mathbb{R}^{N}$, $p>0$ we have
$\mathbf{v}^{\mathsf{T}}\,\tilde{\mathbf{A}}(p)\,\mathbf{v}>0$. The matrix
$\tilde{\mathbf{F}}(p)$ is symmetric and positive definite, hence it is
invertible.
$\Box$

###### Proposition 2.7

If $\mathbf{B}$ is a symmetric matrix-valued Bernstein function and for each
vector $\mathbf{v}\in\mathbb{R}^{N}$ the function
$\mathbf{v}^{\mathsf{T}}\,\mathbf{B}(t)\,\mathbf{v},t>0$, is not identically
zero, then the matrix $\tilde{\mathbf{B}}(p)$ is invertible for every $p>0$.

The proof of Proposition 2.7 is analogous to Proposition 2.6.

Equation (2) is equivalent to the equation

$\left[\mathbf{A}_{1}+\tilde{\mathbf{F}}(p)\right]\,\tilde{\mathbf{X}}(p)=\tilde{\mathbf{R}}(p)$
(10)

If either the matrix $\mathbf{A}_{1}$ is positive definite or the matrix
$\tilde{\mathbf{F}}(p)$ is invertible for $p>0$, then the matrix
$\mathbf{A}_{1}+\tilde{\mathbf{F}}(p)$ is invertible for $p>0$. In this case
the unique solution of (10) is

$\tilde{\mathbf{X}}(p)=\left[\mathbf{A}_{1}+\tilde{\mathbf{F}}(p)\right]^{-1}\,\tilde{\mathbf{R}}(p),\;\;p>0$
(11)

This is true in particular if $\mathbf{F}$ is an $\mathcal{S}_{+}$-valued LICM
function.

For $\mathbf{R}(t)=\mathbf{I}$ ($t>0$), where $\mathbf{I}$ is the identity
operator on $\mathbb{R}^{N}$, $\tilde{\mathbf{R}}(p)=p^{-1}\,\mathbf{I}$ and
$\tilde{\mathbf{X}}(p)=\left[p\,\mathbf{A}_{1}+p\,\tilde{\mathbf{F}}(p)\right]^{-1}$.
We shall show that in this case $\mathbf{X}$ is an LICM function.

For $\mathbf{R}(t)=t\,\mathbf{I},t>0$, we have
$\tilde{\mathbf{R}}(p)=p^{-2}\,\mathbf{I}$ and

$p\,\tilde{\mathbf{X}}(p)=\left[p\,\mathbf{A}_{1}+p\,\tilde{\mathbf{F}}(p)\right]^{-1}$
(12)

We shall show that under these assumptions the solution $\mathbf{X}$ of
equation (6) is an $\mathcal{S}_{+}$-valued Bernstein function.

Similarly, if $\mathbf{A}$ is an $\mathcal{S}_{+}$-valued BF and
$\mathbf{R}(t)=t\,\mathbf{I},t>0$, then $\mathbf{X}$ is an
$\mathcal{S}_{+}$-valued LICM function.

By transposition the results obtained below also apply to equations of the
form

$\mathbf{X}(t)\ast(\mathbf{A}_{1}+\mathbf{F}(t))=\mathbf{R}(t).$

##  3 Main theorem.

Assume that $\mathbf{A}_{1}$ is positive semi-definite symmetric matrix of
rank $N$.

###### Theorem 3.1

If either
(1) $\mathbf{A}_{1}$ is a positive definite symmetric matrix
or else
(2) $\mathbf{F}$ is a non-zero $\mathcal{S}_{+}$-valued LICM function,
satisfying the conditions
$(\ast)$ for each non-zero vector $\mathbf{v}\in\mathbb{R}^{N}$ the function
$\mathbf{v}^{\mathsf{T}}\,\mathbf{F}(t)\,\mathbf{v},t>0$, is not identically
zero, and
$(\ast\ast)$ the limit $\mathbf{F}_{0}:=\lim_{t\rightarrow
0}\mathbf{F}(t)^{-1}$ (13) exists,
then equation (6) with $\mathbf{R}(t)=\mathbf{I}$ has a unique solution
$\mathbf{X}(t)$, where $\mathbf{X}$ is an LICM function in the first case and
$\mathbf{X}(t)=\upsilon(t)\,\mathbf{F}_{0}+\mathbf{G}(t)$, where $\mathbf{G}$
is an $\mathcal{S}_{+}$-valued LICM function, in the second case.

In general the solution of equation (6) $\mathbf{X}\in\mathcal{E}$.

Remark. Concerning Condition $(\ast\ast)$, we begin with the remark that the
matrix $\mathbf{F}(t)$ is invertible for sufficiently large $t$. Indeed, on
account of Condition $(\ast)$ each of its eigenvalues $a_{n}(t)$ ($n=1,\ldots
N$) is a CM function and is positive for some $t_{n}$. Consequently the matrix
$\mathbf{F}(t)$ is invertible for $t>\sup\\{t_{n}\mid n=1,\ldots N\\}$. Hence
it makes sense to inquire whether the limit $\lim_{t\rightarrow
0}\mathbf{F}(t)^{-1}$ exists.

Remark.
The equation $k\ast l=1$ in for locally integrable real-valued functions $k$
and $l$ is known as the Sonine equation [9]. If for a given
$k\in\mathcal{L}^{1}_{\mathrm{loc}}([0,\infty[)$ there is an
$l\in\mathcal{L}^{1}_{\mathrm{loc}}([0,\infty[)$ satisfying the above
equation, then $k$ is called a Sonine function, while $k,l$ are known as a
Sonine pair. Sonine pairs are studied in some detail in [8]. Theorem 3.1
asserts in particular that every LICM function or matrix-valued LICM function
is a Sonine function and in this case the Sonine pair consists of two LICM
functions. For real-valued functions this fact has apparently been discovered
by Kochubei [6].

However, not every Sonine pair consists of LICM functions. A counterexample is
the Sonine pair $k_{\lambda}(t):=t^{-\lambda/2}\,J_{-\lambda}(2t^{1/2})$ with
the Laplace transform $\tilde{k}_{\lambda}(p)=\exp(-1/p)\,p^{\lambda-1}$ ([2]
p. 185 (30)) and $l_{\lambda}(t)t^{(\lambda-1)/2}\,I_{\lambda-1}(2t^{1/2})$
with $\tilde{l}_{\lambda}(p)=\exp(1/p)\,p^{-\lambda-2}$ ([2] p. 197 (18)) for
$\lambda>0$. The function $k_{\lambda}$ changes sign and therefore is not CM,
for example $k_{1/2}(t)=\sqrt{2/\uppi}\,\cos(2t^{1/2})/t^{3/4}$.

The following matrix-valued Sonine pairs are of particular interest:

  1. 1.

$k(t)\,\mathbf{K}_{0}$ and $l(t)\,\mathbf{K}_{0}^{\;-1}$, where $k,l$ are a
Sonine pair of LICM functions;

  2. 2.

$\mathrm{diag}\\{k_{n}(t),n=1,...m\\}$ and
$\mathrm{diag}\\{l_{n}(t),n=1,...m\\}$, where $(k_{n},l_{n})$ are Sonine pairs
of LICM functions for $n=1,\ldots,m$.

Many LICM functions are known [7], but it is often more difficult to find the
other member of the Sonine pair. The simplest Sonine pair of CM functions is
$k(t)=t^{\alpha-1}/\Gamma(\alpha)$ and $l(t)=t^{-\alpha}/\Gamma(1-\alpha)$,
$0<\alpha<1$. Using the Laplace transforms
$\tilde{k}(p)=(p+\lambda)^{-\alpha}$ and $\mathcal{L}[\Gamma(-\alpha,\lambda
t)](p)=\Gamma(-\alpha)\,\lambda^{-\alpha}\left[\lambda^{\alpha}-(\lambda+p)^{\alpha}\right]/p$
one gets another pair $k(t)=t^{\alpha-1}\,\mathrm{e}^{-\lambda
t}/\Gamma(\alpha)$, $\lambda>0$, with
$l(t)=\lambda^{\alpha}\,\left[1-\Gamma(-\alpha,\lambda
t)/\Gamma(-\alpha)\right]$, $\lambda\geqslant 0$, $0<\alpha<1$.

It is also interesting that for an arbitrary analytic function $k(t)$ there is
another analytic function $l(t)$ such that $k(t)\,t^{\alpha-1}/\Gamma(\alpha)$
and $l(t)\,t^{-\alpha}/\Gamma(1-\alpha)$ are a Sonine pair and there is an
algorithm for calculating the power series of $l(t)$ given the power series
for $k(t)$ [8, 11].

Proof of Theorem 3.1.

The Laplace transform $\tilde{\mathbf{F}}(p)$ is a symmetric positive definite
matrix for every $p>0$. For $p>0$ equation (6) is equivalent equation
$\tilde{\mathbf{X}}(p)=\left[p\,(\mathbf{A}_{1}+\tilde{\mathbf{F}}(p))\right]^{-1}$.
The inverse on the right-hand side exists for $p>0$ if $\mathbf{A}_{1}$ is
positive definite or else in view of Condition ($\ast$) and Proposition 2.6.
The right-hand side of the last equation is the algebraic inverse of a matrix-
valued CBF, hence it is a matrix-valued Stieltjes function of the form

$\mathbf{K}(p)=\mathbf{C}+\int_{[0,\infty[}(p+r)^{-1}\,\mathbf{H}(r)\,\mu(\mathrm{d}r).$
(14)

where $\mathbf{C}\geqslant 0$, $\mathbf{H}$ is a measurable symmetric matrix-
valued function bounded $\mu$-almost everywhere and $\mu$ is a Borel measure
satisfying inequality (9) (Theorem B.2). The second term on the right-hand
side of equation (14) is a Laplace transform $\tilde{\mathbf{G}}(p)$ of the
LICM matrix-valued function

$\mathbf{G}(t):=\int_{[0,\infty[}\mathrm{e}^{-rt}\,\mathbf{H}(r)\,\mu(\mathrm{d}r)$
(15)

By Proposition 2.3

$\mathbf{C}=\lim_{p\rightarrow\infty}\mathbf{K}(p)=\left[\lim_{p\rightarrow\infty}\,(p\,\mathbf{A}_{1}+p\,\tilde{\mathbf{F}}(p))\right]^{-1}$

Again by Proposition 2.3, if $\mathbf{A}_{1}>0$, then $\mathbf{C}=0$. If
$\mathbf{A}_{1}=0$, then in view of Condition $(\ast\ast)$
$\mathbf{C}=\mathbf{F}_{0}$.

Applying the inverse Laplace transformation to (14) we conclude that
$\mathbf{X}=\mathbf{G}$ if $\mathbf{A}_{1}>0$,
$\mathbf{X}(t)=\upsilon(t)\,\mathbf{F}_{0}+\mathbf{G}(t)$ if
$\mathbf{A}_{1}=0$.

Uniqueness of the solution $\mathbf{X}(p)$ follows from the fact that in view
of the invertibility of the matrix $\mathbf{A}_{1}+\tilde{\mathbf{F}}(p)$ the
equation $(\mathbf{A}_{1}+\tilde{\mathbf{F}}(p))\,\tilde{\mathbf{X}}(p)=0$ for
$p>0$ implies that $\tilde{\mathbf{X}}(p)=0$ for $p\geqslant 0$.

$\Box$

###### Theorem 3.2

If $\mathbf{A}_{1}=0$ and $\mathbf{F}$ is a singular LICM matrix-valued
function, then the solution $\mathbf{X}$ of equation (6) is a singular LICM
function $\mathbf{G}$.

Proof.
Since $\mathbf{A}_{1}=0$, the solution of equation (6) is an LICM matrix-
valued function $\mathbf{G}$ and

$\lim_{t\rightarrow
0}\mathbf{G}(t)^{-1}=\lim_{p\rightarrow\infty}\left(p\,\tilde{\mathbf{G}}(p)\right)^{-1}=\lim_{p\rightarrow\infty}\tilde{\mathbf{F}}(p)=\lim_{t\rightarrow
0}\int_{0}^{t}\mathbf{F}(s)\,\mathrm{d}s=0,$

because the function $\mathbf{F}$ is locally integrable.  $\Box$

###### Corollary 3.1

For $n\in\mathbb{N}$ the convolution equation (6) with
$\mathbf{R}(t)=(t^{n}/n!)\,\mathbf{I}$ and an LICM matrix-valued function
$\mathbf{F}$ satisfying Condition $(\ast)$ has a unique solution $\mathbf{X}$
which is an $n$-fold indefinite integral of an element of $\mathcal{E}$.

In particular, for $n=1$ the solution $\mathbf{X}$ is a matrix-valued
Bernstein function and $\mathbf{X}(0)=0$ if $\mathbf{A}_{1}>0$, while
$\mathbf{X}(0)=\mathbf{F}_{0}$ if $\mathbf{A}_{1}=0$, with $\mathbf{F}_{0}$
given by equation (13).

For the proof of the last statement note that $\lim_{t\rightarrow
0}\int_{0}^{t}\mathbf{F}(s)\,\mathrm{d}s=0$ because $\mathbf{F}$ is integrable
in a neighborhood of 0.

Here is a complementary result for the last statement:

###### Corollary 3.2

Let $\mathbf{B}$ be an $\mathcal{S}_{+}$-valued Bernstein function satisfying
the condition
$(\ast\ast\ast)$ For every $\mathbf{v}\in\mathbb{R}^{N}$ the function
$\mathbf{v}^{\mathsf{T}}\,\mathbf{B}(t)\,\mathbf{v}$, $t>0$, is not constant.
The equation

$\mathbf{B}\ast\mathbf{X}=t\,\mathbf{I}$ (16)

has a unique solution $\mathbf{X}$ and this solution is an
$\mathcal{S}$-valued LICM function.

Proof.

Differentiating equation (16) with respect to $t$ one gets equation (6) with
$\mathbf{R}(t)=\mathbf{I}$ and $\mathbf{F}=\mathbf{B}^{\prime}$, an LICM
function satisfying Condition $(\ast)$. The thesis then follows from Theorem
3.1.

$\Box$

##  4 Anisotropic generalized fractional derivatives (GFD).

The term ”derivative” has been improperly applied to fractional derivatives
although they do not satisfy the Leibniz property, which is part of the
definition of a derivative. Fractional derivatives have however provided
useful tools for constructing equations which in some sense interpolate
between differential equations of varying orders. We shall now show that
similar operators can be constructed for a kind of anisotropic generalized
fractional derivative (GFD) which might be useful to construct anisotropic
relaxation equations and for other purposes.

The first application of this result is the solution of a convolution equation

$\mathbf{F}(t)\ast\mathbf{v}(t)=\mathbf{f}(t)$ (17)

where $\mathbf{F}$ is a given matrix-valued function. Since

$\mathbf{X}(t)\ast\mathbf{F}(t)=\mathbf{I},$ (18)

we have

$\int_{0}^{t}\mathbf{v}(t)\,\mathrm{d}t=\mathbf{X}(t)\ast\mathbf{f}(t)$

and therefore

$\mathbf{v}(t)=\mathrm{D}[\mathbf{X}\ast\mathbf{f}(t)]$ (19)

The LICM matrix-valued function $\mathbf{F}(t)$ can have a singularity at 0
such that for every $\mathbf{v}\in\mathbb{R}^{N}$ the limit
$\lim_{t\rightarrow
0}\,\mathbf{v}^{\mathsf{T}}\,\mathbf{F}(t)\,\mathbf{v}=\infty$. In this case
$\mathbf{F}_{0}=0$, where $\mathbf{F}_{0}$ is defined by equation (13). We
shall say that the function $\mathbf{F}$ is singular if $\mathbf{F}_{0}=0$.

If $\mathbf{F}$ is a singular LICM matrix-valued function, then by Theorem 3.1
the solution $\mathbf{X}=\mathbf{G}$ of the Sonine equation
$\mathbf{F}\ast\mathbf{X}=\mathbf{I}$ is an LICM function and we can define
the $\mathbf{F}$-derivative by the formula

$\mathrm{D}_{\mathbf{F}}\,\mathbf{v}(t)=\mathrm{D}\left[\mathbf{F}(t)\ast\mathbf{v}(t)\right]-\mathbf{F}(t)\,\mathbf{v}(0)\equiv\mathbf{F}\ast\mathrm{D}\mathbf{v}$
(20)

for every absolutely continuous function
$\mathbf{v}:\;[0,\infty[\rightarrow\mathbb{R}^{N}$. If $\mathbf{X}=\mathbf{G}$
(a singular LICM function) is the solution of the convolution equation (6)
with $\mathbf{R}(t)=\mathbf{I}$, then the $\mathbf{F}$-integral operator is
defined by the formula

$\mathrm{J}_{\mathbf{F}}\,\mathbf{v}(t):=\mathbf{G}\ast\mathbf{v}(t).$ (21)

It follows from Theorem 3.2 that the function $\mathbf{G}$ is singular.

We then have

###### Theorem 4.1

Let $\mathbf{F}$ be a singular LICM matrix-valued function.

The following relations hold

$\displaystyle\mathrm{J}_{\mathbf{F}}\,\mathrm{D}_{\mathbf{F}}\,\mathbf{w}(t)=\mathbf{w}(t)-\mathbf{w}(0)\;\;\mathrm{for}\;\;\mathbf{w}\in
AC([0,\infty[)$ (22)
$\displaystyle\mathrm{D}_{\mathbf{F}}\,\mathrm{J}_{\mathbf{F}}\,\mathbf{v}(t)=\mathbf{v}(t)\;\;\mathrm{for}\;\;\mathbf{v}\in\mathcal{L}^{1}_{\mathrm{loc}}([0,\infty[)$
(23)

Proof
(1) The identity $\mathbf{G}\ast\mathbf{F}=\mathbf{1}$ implies that

$(\mathrm{J}_{\mathbf{F}}\,\mathrm{D}_{\mathbf{F}}\,\mathbf{w})(t)=\int_{0}^{t}\mathbf{G}(s)\left.\frac{\mathrm{d}}{\mathrm{d}\tau}\int_{0}^{\tau}\mathbf{F}(\tau-r)\,\mathbf{w}(r)\,\mathrm{d}r\right|_{\tau=t-s}\,\mathrm{d}s-\mathbf{w}(0)$

The first term on the right-hand side equals

$\int_{0}^{t}\mathbf{G}(s)\frac{\mathrm{d}}{\mathrm{d}t}\int_{0}^{t-s}\mathbf{F}(t-s-r)\,\mathbf{w}(r)\,\mathrm{d}r\,\mathrm{d}s=\frac{\mathrm{d}}{\mathrm{d}t}(\mathbf{G}\ast\mathbf{F}\ast\mathbf{w})(t)=\frac{\mathrm{d}}{\mathrm{d}t}(\mathbf{I}\ast\mathbf{w})(t)=\mathbf{w}(t)$

q.e.d.

(2) Let $\mathbf{w}:=\mathbf{G}\ast\mathbf{v}$. On account of the identity
$\mathbf{F}\ast\mathbf{G}=\mathbf{I}$

$\mathrm{D}_{\mathbf{F}}\,\mathrm{J}_{\mathbf{F}}\,\mathbf{v}=\frac{\mathrm{d}}{\mathrm{d}t}(\mathbf{F}\ast\mathbf{G}\ast\mathbf{v})(t)-\mathbf{F}(t)\,\mathbf{w}(0)=\mathbf{v}(t)-\mathbf{F}(t)\,\mathbf{w}(0)$

It remains to prove that $\mathbf{w}(0)=0$. $\mathbf{G}$ is an LICM function,
hence it has the form (15) with $\mu$ satisfying equation (9) and
$|\mathbf{H}(r)\,|\leqslant 1$. Hence

$|\mathbf{w}(t)|^{2}\leqslant\left[\int_{0}^{t}\int_{[0,\infty[}\mathrm{e}^{-rs}\,\mu(\mathrm{d}r)\,\mathrm{d}s\right]\,\int_{0}^{t}|\mathbf{v}(t-s)|\,\mathrm{d}s$

For $t\leqslant 1$ the second factor is bounded from above by a constant

$\int_{0}^{1}|\mathbf{v}(s)|\,\mathrm{d}s<\infty$

because $\mathbf{v}$ is assumed locally integrable. The first factor equals

$\int_{[0,\infty[}\frac{1-\mathrm{e}^{-rt}}{r}\,\mu(\mathrm{d}r)$ (24)

From the inequality $\mathrm{e}^{x}-1\leqslant x\,\mathrm{e}^{x}$ ($x\geqslant
0$) follows the inequality $1-\mathrm{e}^{-x}\leqslant x$. We shall apply this
inequality for $r\in[0,\infty[$, noting that $\mu([0,1])<\infty$ because of
(9) with the inequality $1\leqslant 2/(1+r)$ valid for $r\leqslant 1$. For
$r>1$ we shall note that $1/r\leqslant 2/(1+r)$. Hence expression (24) is
bounded by

$t\,\mu([0,1])+2\int_{]1,\infty[}\left(1-\mathrm{e}^{-rt}\right)\,(1+r)^{-1}\,\mu(\mathrm{d}r),$

which tends to 0 as $t\rightarrow 0$ on account of (9) and the Lebesgue
Dominated Convergence Theorem. Thus $\mathbf{w}(0)=0$ and the theorem has been
proved.

$\Box$

The new derivative concept provides a new approach to modeling stress
relaxation in anisotropic and non-linear viscoelastic media. A possible
relaxation equation could have the form

$\mathrm{D}_{\mathbf{F}}\,\Sigma=\mathbf{K}(\Sigma,E)$ (25)

where $\mathbf{K}$ is a rank-2 tensor-valued function of two rank-2 tensor-
valued arguments.

In view of equation (20)2 and (18) equation (25) is equivalent to the equation

$\Sigma=\Sigma(0)+\mathbf{X}\ast\,\mathbf{K}(\Sigma,E)$ (26)

In the scalar case Theorem 4.1 applies only to (weakly) singular CM kernels
$\mathbf{F}$ such as $t^{-\alpha}/\Gamma(1-\alpha)$.

##  5 Application to anisotropic linear viscoelasticity.

The relaxation modulus $F_{jklm}(t)$ and the creep function $C_{jklm}(t)$ in
3-dimensional linear viscoelasticity are defined by the two constitutive laws
which are assumed equivalent to each other

$\displaystyle\Sigma_{jk}=N_{jklm}\,\mathrm{D}{E}_{lm}+F_{jklm}\,\ast\,\mathrm{D}{E}_{lm},\;j,k=1,\ldots
3$ (27) $\displaystyle
E_{jk}=C_{jklm}\,\ast\,\mathrm{D}{\Sigma}_{lm}\;j,k=1,\ldots 3\ $ (28)

(summation over repeated indices is assumed) where the symmetric tensors
$\Sigma_{kl}$ and $E_{kl}$ denote the stress and strain tensors, respectively.
We assume the usual symmetries

$\displaystyle F_{ijkl}=F_{jikl}=F_{klij}$ $\displaystyle
C_{ijkl}=C_{jikl}=C_{klij}$ (29) $\displaystyle N_{ijkl}=N_{jikl}=N_{klij}$
$\displaystyle\mathrm{for}\;\;i,j,k,l=1\,\ldots 3$

Equivalence of the two constitutive equations is ensured by the relation

$N_{jklm}\,C_{jklm}+F_{jklm}\ast
C_{lmrs}=t\,(\delta_{jr}\,\delta_{ks}+\delta_{js}\,\delta_{kr}),\;\;j,k,r,s=1,\ldots,3,\;t>0$
(30)

The first term on the left-hand side represents a Newton viscosity component.
It is assumed to satisfy the inequalities $N_{jklm}\,e_{jk}\,e_{lm}\geqslant
0$ for every symmetric tensor $e_{kl}$.

Defining the index $I$, $1\leqslant I\leqslant 6$, $I=k$ for the pair $kk$,
$1\leqslant k\leqslant d$ and $I=m$ for the pair $kl$, $k\neq l,m$, $l\neq m$,
$R_{IJ}=f(I)\,f(J)\,R_{IIJJ}$, with $f(I)=1$ if $1\leqslant I\leqslant 3$ and
$f(I)=\sqrt{2}$ if $4\leqslant I\leqslant 6$. In this notation equation (30)
assumes the form of a matrix convolution of two symmetric 6 $\times$ 6 matrix-
valued functions

$N_{IJ}\,C_{JK}+F_{IJ}\,\ast C_{JK}=t\,\delta_{JK}\;\;J,K=1,\ldots 6$ (31)

In the Laplace domain equation (31) is equivalent to

$N_{IJ}\,\widetilde{C_{JK}}+\widetilde{F_{IJ}}\,\widetilde{C_{JK}}=p^{-2}\,\delta_{JK},\;\;J,K=1,\ldots
6$ (32)

The matrix-valued function $F_{IJ}$ is LICM if for every
$\mathbf{v}\in\mathbb{R}^{6}$ the function $t\rightarrow
v_{I}\,v_{J}\,F_{IJ}(t)$ is LICM. The above statement is equivalent to
$e_{ij}\,e_{kl}\,F_{ijkl}(t)$ being LICM for every symmetric 3 $\times$ 3
matrix $e_{ij}$.

We shall use the notation $\mathbf{N}$, $\mathbf{R}$, $\mathbf{C}$ for the 6
$\times$ 6 matrices $\mathbf{N}_{IJ}$, $R_{IJ}$ and $C_{IJ}$. The inequalities
$\mathbf{N}\geqslant 0$, $\mathbf{N}>0$ are short-hand for the inequalities
$N_{ijkl}\,e_{ij}\,e_{kl}\geqslant 0$ for an arbitrary rank-2 tensor $e_{kl}$
and $N_{ijkl}\,e_{ij}\,e_{kl}>0$ for an arbitrary non-zero rank-2 tensor
$e_{kl}$, respectively.

In this notation equation (32) assumes the form

$\mathbf{N}\,\tilde{\mathbf{C}}(p)+\tilde{\mathbf{F}}(p)\,\tilde{\mathbf{C}}(p)=p^{-2}\,\mathbf{I}$
(33)

###### Theorem 5.1

1\. If the matrix $\mathbf{N}$ is symmetric positive semi-definite and
$\mathbf{F}$ is an LICM matrix-valued function satisfying the following
conditions:
$(\ast\ast_{1})$ for every non-zero vector $\mathbf{v}\in\mathbb{R}^{N}$ the
function $\mathbf{v}^{\mathsf{T}}\,\mathbf{F}(t)\,\mathbf{v}$ does not vanish
identically,
$(\ast\ast_{2})$ $\mathbf{F}_{0}:=\lim_{t\rightarrow 0}\,\mathbf{F}(t)^{-1}$
exists and the symmetry relations (5)1 are satisfied, then equation (30) has a
single solution $C_{ijkl}(t)$. The function $C_{ijkl}(t)$ is a rank-4 tensor
valued Bernstein function satisfying (5).
2\. If $C_{ijkl}(t)$ is a rank-4 tensor-valued Bernstein function such that
$(\ast\ast_{3})$ for every non-zero symmetric rank-2 tensor $e_{kl}$ the
function $C_{ijkl}(t)\,e_{ij}\,e_{kl}$ is not constant and $C_{ijkl}$
satisfies the symmetry relations (5)2, then equation (30) has a unique
solution $\left(\mathbf{N},\mathbf{F}\right)$. The function $\mathbf{F}$ is a
rank-4 tensor valued LICM function satisfying (5), while $\mathbf{N}$ is a
rank-4 tensor satisfying (5) and $\mathbf{N}\geqslant 0$ for every rank-2
tensor $e_{kl}$.

The first part of the theorem follows from Corollary 3.1.
The second part follows from Corollary 3.2.

###### Theorem 5.2

Under the hypotheses of Theorem 5.1 the following relations hold

  1. 1.

If $\mathbf{N}>0$, then $\mathbf{C}(0)=0$ and $\lim_{t\rightarrow
0}\mathbf{C}^{\prime}(t)=\mathbf{N}^{-1}$;

  2. 2.

If $\mathbf{N}=0$, then $\mathbf{C}(0)=\mathbf{F}_{0}$.

  3. 3.

The limit $\lim_{t\rightarrow\infty}\mathbf{F}(t)=:\mathbf{F}_{\infty}$ always
exists and is positive-semi-definite.

  4. 4.

If $\mathbf{F}_{\infty}$ is invertible, then
$\lim_{t\rightarrow\infty}\mathbf{C}(t)$ exists and

$\lim_{t\rightarrow\infty}\mathbf{C}(t)=\mathbf{F}_{\infty}^{\;-1}$ (34)

Proof.

$\tilde{\mathbf{R}}(p)=\mathbf{N}+\int_{[0,\infty[}(r+p)^{-1}\,\mathbf{G}(r)\,\mu(\mathrm{d}r),$

where the Borel measure $\mu$ satisfies equation (9) and
$|\mathbf{G}(r)|\leqslant 1$ $\mu$-almost everywhere, and $\lim_{t\rightarrow
0}\mathbf{C}(t)=\lim_{p\rightarrow\infty}\left[p^{-1}\,\tilde{\mathbf{R}}(p)^{-1}\right]$,
hence

$\lim_{t\rightarrow
0}\mathbf{C}(t)=\lim_{p\rightarrow\infty}\left\\{p^{-1}\,\left[\mathbf{N}+\int_{[0,\infty[}(r+p)^{-1}\,\mathbf{G}(r)\,\mu(\mathrm{d}r)\right]^{-1}\right\\}$
(35)

The second term in the square brackets is the Laplace transform of an LICM
function $\mathbf{F}$, defined by equation (15). By Proposition 2.3

$\lim_{p\rightarrow\infty}\left[\mathbf{N}+\int_{[0,\infty[}(r+p)^{-1}\,\mathbf{G}(r)\,\mu(\mathrm{d}r)\right]=\mathbf{N}.$

Thus if $\mathbf{N}$ is invertible, then
$\lim_{p\rightarrow\infty}\left[\mathbf{N}+\int_{[0,\infty[}(r+p)^{-1}\,\mathbf{G}(r)\,\mu(\mathrm{d}r)\right]^{-1}$
exists and equals $\mathbf{N}^{-1}$. It follows from equation (35) that
$\lim_{t\rightarrow 0}\mathbf{C}(t)=0$ in this case.

Furthermore, under the same assumption

$\lim_{t\rightarrow
0}\mathbf{C}^{\prime}(t)=\lim_{p\rightarrow\infty}[p^{2}\,\tilde{\mathbf{C}}(p)]=\lim_{p\rightarrow\infty}\left[\mathbf{N}+\int_{]0,\infty[}(r+p)^{-1}\,\mathbf{G}(r)\,\mu(\mathrm{d}r)\right]^{-1}=\mathbf{N}^{-1}$

If $\mathbf{N}=0$ then

$\lim_{t\rightarrow
0}\mathbf{C}(t)=\lim_{p\rightarrow\infty}[p\,\tilde{\mathbf{C}}(p)]=\lim_{p\rightarrow\infty}\left[p\,\tilde{\mathbf{F}}(p)\right]^{-1}$

The limit of the expression on the right-hand side is $\lim_{t\rightarrow
0}\mathbf{F}(t)^{-1}=\mathbf{F}_{0}$.

Finally,

$\lim_{t\rightarrow\infty}\mathbf{C}(t)=\lim_{p\rightarrow
0}[p\,\tilde{\mathbf{C}}(p)]=\lim_{p\rightarrow
0}\left[p\,\mathbf{N}+p\int_{[0,\infty[}(p+r)^{-1}\mathbf{G}(r)\,\mu(\mathrm{d}r)\right]^{-1}$

If the limit $\lim_{p\rightarrow
0}\left[p\int_{[0,\infty[}(p+r)^{-1}\mathbf{G}(r)\,\mu(\mathrm{d}r)\right]=\lim_{t\rightarrow\infty}\mathbf{F}(t)$
exists and is invertible then equation (34) is satisfied.

$\Box$

We now examine the reverse direction from creep tests to some parameters of
the relaxation function.

###### Proposition 5.1

1\. If $\mathbf{C}(t)\not\equiv 0$ then the function $\mathbf{C}(t)^{-1}$ has
a limit $\mathbf{G}_{1}\geqslant 0$ for $t\rightarrow\infty$.
2\. If $\mathbf{G}_{1}>0$, then
$\lim_{t\rightarrow\infty}\mathbf{C}(t)=\mathbf{G}_{1}^{\;-1}$.
3\. The derivative $\mathbf{C}^{\prime}(t)$ has a limit $\mathbf{G}_{2}$ for
$t\rightarrow\infty$. The limit $\lim_{t\rightarrow 0}\mathbf{C}^{\prime}(t)$
exists, possibly infinite.
4\. If a $\mathbf{C}(t)$ has a finite limit for $t\rightarrow\infty$, then
$\mathbf{G}_{2}=0$.

Proof.
Ad 1. The function $\mathbf{C}(t)^{-1}$ is positive semi-definite and non-
increasing, hence the limit $\mathbf{G}_{1}$ exits and is positive semi-
definite.
Ad 2. It follows from the continuity of algebraic inverse at invertible
matrices.
Ad 3. The function $\mathbf{C}^{\prime}$ is LICM, hence it is positive semi-
definite and non-increasing.
Ad 4. Suppose that $\mathbf{G}_{2}>0$ and $\mathbf{C}(t)$ has a finite limit
$\mathbf{C}_{\infty}$ for $t\rightarrow\infty$. $\mathbf{C}^{\prime}$ is non-
increasing, hence $\mathbf{C}(t)\geqslant\mathbf{C}(1)+(t-1)\,\mathbf{G}_{2}$.
For sufficiently large $t$ the value of $\mathbf{C}(t)$ is larger than
$\mathbf{C}_{\infty}$, which leads to a contradiction.

$\Box$

###### Theorem 5.3

If $\mathbf{C}$ is a $\mathcal{S}_{+}$-valued Bernstein function, and
$(\ast\ast\ast)$ for each non-zero vector $\mathbf{v}\in\mathbb{R}^{6}$ the
function $C_{IJ}(t)\,v_{I}\,v_{J}$ is not identically zero,
then there is an $\mathcal{S}_{+}$-valued LICM function $\mathbf{F}$ and
$\mathbf{N}\in\mathcal{S}_{+}$ such that the pair $(\mathbf{N},\mathbf{F}))$
satisfies equation (31).

Moreover
1\. If $\mathbf{C}(0)>0$ then $\mathbf{N}=0$ and $\lim_{t\rightarrow
0}\mathbf{F}(0)=\mathbf{C}(0)^{-1}$.
2\. If $\mathbf{C}(0)=0$, while $\mathbf{C}^{\prime}_{0}:=\lim_{t\rightarrow
0}\mathbf{C}^{\prime}(t)$ is finite and has an inverse then
$\mathbf{N}=\mathbf{C}^{\prime}(0)^{-1}$.
3.

$\lim_{t\rightarrow\infty}\mathbf{F}(t)=\left\\{\begin{array}[]{cl}0&\mbox{if
$\lim_{t\rightarrow\infty}\mathbf{C}^{\prime}(t)>0$}\\\
\lim_{t\rightarrow\infty}\mathbf{C}(t)^{-1}&\mbox{if
$\lim_{t\rightarrow\infty}\mathbf{C}^{\prime}(t)=0$}\end{array}\right.$

Proof.
The derivative $\mathbf{C}^{\prime}(t)$ is an LICM function and
$\mathbf{C}^{\prime}(t)=\mathbf{B}+\mathbf{Q}(t)$, where
$\mathbf{B}=\lim_{t\rightarrow\infty}\mathbf{C}^{\prime}(t)$, $\mathbf{Q}$ is
LICM and $\lim_{t\rightarrow\infty}\mathbf{Q}(t)=0$. Hence

$\mathbf{C}(t)=\mathbf{A}+t\,\mathbf{B}+\int_{0}^{t}\mathbf{Q}(s)\,\mathrm{d}s.$

where the matrices $\mathbf{A}$ and $\mathbf{B}$ are symmetric and positive
semi-definite. We note that $\mathbf{A}=\mathbf{C}(0)$ and, by Proposition
2.3, $\mathbf{B}=\lim_{t\rightarrow\infty}\mathbf{C}^{\prime}(t)$.

In view of Assumption ($\ast\ast\ast$) the matrix $\tilde{\mathbf{C}}(p)$ has
an inverse for every $p>0$. Since

$p\,\tilde{\mathbf{C}}(p)=\mathbf{A}+p^{-1}\,\mathbf{B}+\tilde{\mathbf{Q}}(p)$

is a Stieltjes function, its algebraic inverse $p\,\tilde{\mathbf{R}}(p)$ is a
CBF. Hence

$p\,\tilde{\mathbf{R}}(p)=p\,\mathbf{N}+p\int_{[0,\infty[}(p+r)^{-1}\,\mathbf{H}(r)\,\mu(\mathrm{d}r)$

where $\mathbf{N}$ is symmetric positive semi-definite $\mu$ is a Borel
measure satisfying (9) and $\mathbf{H}(r)$ a bounded symmetric function for
$\mu$-almost all $r\in[0,\infty[$.

By Proposition 2.3
$\mathbf{N}=\lim_{p\rightarrow\infty}\tilde{\mathbf{R}}(p)$. On the other hand

$\lim_{p\rightarrow\infty}\tilde{\mathbf{R}}(p)=\lim_{p\rightarrow\infty}\left[p^{2}\,\tilde{\mathbf{C}}(p)\right]^{-1}=\lim_{p\rightarrow\infty}\left[p\,\mathbf{A}+\mathbf{B}+p\,\tilde{\mathbf{Q}}(p)\right]^{-1}=\\\
=\left\\{\begin{array}[]{cl}0&\mathbf{A}>0\\\
\left[\mathbf{B}+\mathbf{Q}(0)\right]^{-1}&\mathbf{A}=0,\left[\mathbf{B}+\mathbf{Q}(0)\right]>0\end{array}\right.$
(36)

where we note that $\mathbf{B}+\mathbf{Q}(0)=\mathbf{C}^{\prime}(0)$.

Finally

$\lim_{t\rightarrow\infty}\mathbf{F}(t)=\lim_{p\rightarrow\infty}\left[p\,\tilde{\mathbf{R}}(p)\right]=\left\\{\begin{array}[]{cl}0&\mathbf{B}>0\\\
\left[\mathbf{A}+\int_{0}^{\infty}\mathbf{Q}(t)\,\mathrm{d}t\right]^{-1}&\mathbf{B}=0\end{array}\right.$

where we note that
$\mathbf{A}+\int_{0}^{\infty}\mathbf{Q}(t)\,\mathrm{d}t=\lim_{t\rightarrow\infty}\mathbf{C}(t)$
if the limit on the right-hand side exists.
$\Box$

Theorem 5.1 implies that setting the Newtonian viscosity coefficient
$\mathbf{N}=0$ has profound consequences for the creep $\mathbf{C}(t)$: the
creep either starts with a jump or vertically from the zero value. On the
other hand the value of $\mathbf{N}$ can be estimated from creep tests as the
inverse of the original creep rate.

In comparison with [3] the results of Section 5 have demonstrated the
inseparability of the Newtonian component of viscoelasticity from viscoelastic
memory effects.

##  6 Conclusions.

We have demonstrated a particular role of LICM kernels in two classes of
convolution equations and utility of the concepts of CBFs and Stieltjes
derivatives in the study of existence problems for these equations. A
particular class of convolution equations studied here are fundamental in
linear viscoelasticity.

Another convolution equation has allowed us to define a class of anisotropic
generalized fractional derivatives associated with matrix-valued LICM kernels.
We have also shown that the kernel appearing in the definition of anisotropic
GFD and the associated anisotropic fractional integrals should be singular at
0.

##  Appendix A A remark on the convolution algebra.

For our purposes it is important that the convolution algebra has a unity. The
unity is not the convolution with a function. Hence the convolution algebra
must include Borel measures. The convolution $\rho\ast\nu$ of two measures
$\rho$ and $\nu$ defined on $[0,\infty[$ is defined as the Borel measure
$\lambda$ satisfying the identity

$\int_{[0,\infty[}f(r)\,\mathrm{d}\lambda(\mathrm{d}r)=\int_{[0,\infty[}\int_{[0,\infty[}f(r+s)\,\rho(\mathrm{d}r)\,\nu(\mathrm{d}s)$

for every continuous function $f$ with compact support.

This definition is easily extended to matrix-valued measures.

For our purposes the convolution algebra has to involve only Borel measures of
the form $u\,\mathbf{C}+\mathbf{F}(t)\,\mathrm{d}t$, where the unity is a
measure defined in Section 2.

##  Appendix B Matrix-valued Stieltjes functions and Complete Bernstein
functions.

We shall now use some results from Appendix B of [5].

A matrix-valued Stieltjes function $\mathbf{Y}(p)$ has the following integral
representation:

$\displaystyle\mathbf{Y}(p)=\mathbf{B}+\int_{[0,\infty[}(p+r)^{-1}\,\mathbf{H}(r)\,\mu(\mathrm{d}r)$
(37)

where $\mathbf{B}\in\mathcal{S}_{+}$, $\mu$ is a Borel measure on $[0,\infty[$
satisfying (9) and $\mathbf{H}(r)$ is an $\mathcal{S}_{+}$-valued function
defined and bounded $\mu$-almost everywhere on $[0,\infty[$.
Conversely, any matrix-valued function with the integral representation (37)
is an $\mathcal{S}_{+}$-valued Stieltjes function.

###### Theorem B.1

Every matrix-valued Stieltjes function is the Laplace transform of an element
of $\mathcal{E}$.

Proof.

The Laplace transform of the $\mathcal{S}_{+}$-valued LICM function
$\mathbf{A}(t)$ (equation (8)) is given by the equation

$\tilde{\mathbf{F}}(p)=\int_{[0,\infty[}(p+r)^{-1}\,\mathbf{H}(r)\,\mu(\mathrm{d}r)$
(38)

where $\mu$, $\mathbf{H}$ satisfy the same conditions as in (37).

The second term on the right-hand side of equation (37) involves a double
Laplace transform, hence it equals

$\mathbf{V}(t):=\int_{0}^{\infty}\mathrm{e}^{-pt}\left[\int_{0}^{\infty}\mathrm{e}^{-rt}\,\mathbf{H}(r)\,\mu(\mathrm{d}r)\right]\mathrm{d}t$

where the Borel measure $\mu$ satisfies inequality (9). The inner integral
represents a general matrix-valued LICM $\mathbf{F}(t)$. Thus $\mathbf{V}(t)$
is the Laplace transform of a general matrix-valued LICM and $\mathbf{Y}(p)$
is the Laplace transform of a general element of $\mathcal{E}$.

$\Box$

An $\mathcal{S}_{+}$-valued CBF $\mathbf{Z}(p)$ has the following integral
representation:

$\displaystyle\mathbf{Z}(p)=p\,\mathbf{B}+p\int_{[0,\infty[}(p+r)^{-1}\,\mathbf{H}(r)\,\nu(\mathrm{d}r)$
(39)

where $\mathbf{B}\in\mathcal{S}_{+}$, $\nu$ is a Borel measure on $[0,\infty[$
satisfying (9) and $\mathbf{H}(r)$ is an $\mathcal{S}_{+}$-valued function
defined $\nu$-almost everywhere on $[0,\infty[$.

Conversely, any $\mathcal{S}_{+}$-valued function with the integral
representation (39) is a $\mathcal{S}_{+}$-valued CBF.

It follows immediately that the the function $p^{-1}\,\mathbf{Z}(p)$, where
$\mathbf{Z}$ is an $\mathcal{S}_{+}$-valued CBF function, is an
$\mathcal{S}_{+}$-valued Stieltjes function.

We quote Lemma 1 in Appendix B of op. cit. in the form of the following
theorem

###### Theorem B.2

If $\mathbf{Z}(p)$ is an $\mathcal{S}_{+}$-valued CBF and does not vanish
identically, then $\mathbf{Z}(p)^{-1}$ is an $\mathcal{S}_{+}$-valued
Stieltjes function.

Conversely, if $\mathbf{Y}(p)$ is an $\mathcal{S}_{+}$-valued function does
not vanish identically then $\mathbf{Y}(p)^{-1}$ is a CBF.

## References

  * [1] A. Hanyga, _On solutions of matrix-valued convolution equations, CM-fractional derivatives and their applications in linear and non-linear anisotropic viscoelasticity_. Z. angew. Mathematik Physik, (2019) 70–103.
  * [2] A. Erdélyi, W. Magnus, F. Oberhettinger, and F. Tricomi.  Higher Transcendental Functions.  McGraw-Hill, New York (1953)
  * [3] A. Hanyga and M. Seredyńska, _Relations between relaxation modulus and creep compliance in anisotropic linear viscoelasticity_ , J. of Elasticity 88 (2007) 41–61.
  * [4] A. Hanyga, A simple proof of a duality theorem with applications in scalar and anisotropic viscoelasticity, arxiv: (2018) 1805.07275.
  * [5] A. Hanyga, Wave propagation in anisotropic elasticity, J. of Elasticity 122 (2016) 231–254.
  * [6] A. N. Kochubei, _General fractional calculus, evolution equations and renewal processes_ , arxiv: 1105.1239 (2011).
  * [7] K. S. Miller and S. G. Samko, _Completely monotonic functions_ , Integr. Transf. and Spec. Fun. 12 (2001) 389–402.
  * [8] S. G. Samko and R. P. Cardoso, _Integral equations of the first kind of Sonine type_ , Intern. J. Math. and Math. Sci. 57 (2003) 3609–3632.
  * [9] S. G. Samko, A. A. Kilbas, and O. I. Marichev, _Fractional integrals and derivatives. theory and applications_ , Gordon and Breach, London (1993).
  * [10] R. L. Schilling, R. Song and Z. Vondraček, _Bernstein Functions: Theory and Applications_ , Walter De Gruyter, 2012.
  * [11] J. Wick,  Über eine Integralgleichung vom Abelschen Typ ZAMM 48, 39–41, 1968.


# Fine-Tuning BERT for Sentiment Analysis of Vietnamese Reviews

Quoc Thai Nguyen12, Thoai Linh Nguyen12, Ngoc Hoang Luong12, and Quoc Hung
Ngo12  1University of Information Technology, Ho Chi Minh City, Vietnam
2Vietnam National University, Ho Chi Minh City, Vietnam

###### Abstract

Sentiment analysis is an important task in the field of Nature Language
Processing (NLP), in which users’ feedback data on a specific issue are
evaluated and analyzed. Many deep learning models have been proposed to tackle
this task, including the recently-introduced Bidirectional Encoder
Representations from Transformers (BERT) model. In this paper, we experiment
with two BERT fine-tuning methods for the sentiment analysis task on datasets
of Vietnamese reviews: 1) a method that uses only the [CLS] token as the input
for an attached feed-forward neural network, and 2) another method in which
all BERT output vectors are used as the input for classification. Experimental
results on two datasets show that models using BERT slightly outperform other
models using GloVe and FastText. Also, regarding the datasets employed in this
study, our proposed BERT fine-tuning method produces a model with better
performance than the original BERT fine-tuning method.

Keywords – sentiment analysis, BERT, pre-trained language model, deep
learning.

##  1 Introduction

Nowadays, customers often leave their reviews about various kinds of products
and services, which they have used on a wide range of online platforms, from
shopping sites to social media. Such user experience information is a valuable
source of reference for potential customers to make their buying decisions,
and especially, for related companies to improve their businesses. However, it
is prohibitively cumbersome to manually process such large amounts of data.
Automated sentiment analysis systems are, therefore, essential to study
customers’ subjective opinions from their reviews in an efficient manner. A
fundamental task of such systems is sentiment analysis, in which it is
inferred from the contents of each review whether the reviewer likes or
dislikes the thing being discussed about.

In 2018, Devlin et al. [5] introduces a new language representation model,
namely BERT (Bidirectional Encoder Representations from Transformers). This
model has successfully improved recent works in finding representations of
words in a digital space from their context. In this paper, we will present
two methods for the sentiment analysis of Vietnamese reviews, and both are
based on the BERT language model. Firstly, we implement the classification
method proposed in Jacob Devlin’s study [5]. Secondly, we propose a new method
for text classification that integrates BERT into three classification models.
We aim to exhibit the effectiveness of BERT for the sentiment analysis task
compared to other models. We carry out other fine-tuning BERT methods for
sentiment analysis and select the best model to combine with BERT.

The next section reviews related works for the sentiment analysis task.
Section 3 gives an overview of Word Embedding, Language model and BERT model.
Section 4 describes details of two methods for fine-tuning BERT on sentiment
analysis. The experimental results and analysis with our classification
approach are described in Section 5. Finally, we conclude the paper and give
some future research directions in Section 6.

##  2 Related Work

Sentiment analysis is one of text classification tasks, in which the input
text is classified as sentiment labels, such as “positive” or “negative”
label. There are several studies on this task with machine learning approaches
and lexicon/dictionary approaches (as shown in Table I). In 2002, the first
research to classify the reviews into two groups, positive and negative [3].
Several studies used supervised learning models to solve the task, such as,
Support Vector Machine (SVM) [6], [7], Naïve Bayes [1], [14]. In another
study, they use a dictionary of emotional vocabulary, which indicates whether
a word is positive or negative along with it level [9].

Recently, deep learning approaches allow to learn better representations for
words based on their context. Yoon Kim used Convolution Neural Network (CNN)
[16] for the document classification task, the author process with each
character-based. Mikolov et al. [19] proposed an unsupervised learning
algorithm neural network called “Paragraph vector”, which is similar to the
work in [21]. However, they not using bag-of-words of context words as input,
but an additional matrix of text with each column of matrix is a “Paragraph
vector”.

In Vietnamese text, Duyen et al. use Naïve Bayes, Max Entropy Model and SVM to
classify reviews on the Agoda site [11], which allows users to book hotel
rooms on travel occasions. The results show that the SVM model achieves the
best result. Quan et al. [20] employs a deep learning approach, proposing a
model that combines Long Short-Term Memory (LSTM) and CNN, namely multi-
channel LSTM-CNN for Vietnamese sentiment analysis. The combined model
achieves a performance better than both CNN and LSTM alone. This approach is
similar to [10], which a deep learning model is used for managing negative
comments on social networks. Word vectors are passed through the CNN
component, and the output is then used as the input for the LSTM network to
perform classification.

TABLE I: Approaches for Sentiment Analysis

| Description | Author
---|---|---
English
Lexicon-based | Dataset positive/negative | [9]
Naïve Bayes | Dataset positive/negative |  [14] [1]
SVM | Dataset positive/negative |  [6] [7]
TextCNN | This model is tested 8 Dataset | [16]
Vietnamese
Naïve Bayes | Positive/negative, 73.65% F1-score | [11]
MEM | Positive/negative, 82.6% F1-score | [11]
SVM | Positive/negative, 80.4% F1-score | [11]
CNN-LSTM | Positive/negative/neutral, 87.2%, 59.6 F1-score | [20]
CNN-LSTM | Positive/negative/neutral, 85.5%, 72% F1-score | [10]

In summary, deep learning is currently the state-of-the-art approach for the
field for sentiment analysis. However, Vietnamese resources for NLP tasks are
limited, while popular word representation models, such as Word2Vec or GloVe,
are fixed and not contextually flexible. In this study, we experiment with
fine-tuning BERT to address the sentiment analysis task for Vietnamese
language.

##  3 Background

###  3.1 Word Embedding

Word Embedding is a method of mapping each word into a multi-dimensional real
space, however its size is much smaller than the dictionary size. Several
studies introduced approaches for the word embedding step in NLP tasks. Tomas
Mikolos proposed Word2Vec [21], which is a statistical method for efficiently
learning a standalone word embedding from a text corpus. The global Vectors
for Word Representation (GloVe) algorithm is another method for efficiently
learning word vectors, developed by Pennington et al. [8]. Unlike context-free
techniques (e.g., GloVe, word2vec), Bidirectional Encoder Representation from
Transformer (BERT) is a state-of-the-art language model that generates a
contextual representation for each word, taking into account its neighboring
words [5].

###  3.2 Language Model

The language model is a probability distribution over text sets. The language
model can show how much probability a sentence (or phrase) belongs to a
language. Language models analyze bodies of text data to provide a basis for
their word predictions.

$P(x_{1}….x_{n})$

where $x_{1}$,$x_{2}$,…,$x_{n}$ is the sequence of words that make up a
sentence, and $n$ is length of the sentence (n $>$ 1).

Currently, pre-trained language models are widely used in NLP tasks. These
models have been trained on a very large dataset, integrating many languages
and knowledge. Users can thus use them in many different NLP tasks [5].

###  3.3 BERT

BERT is a multi-layered structure of Bidirectional Transformer encoder layers,
based on the architecture of transformer [2]. BERT uses Bidirectional
Transformer encoders replace Encoders combining Decoders. BERT fine-tuning
tasks do not require Decoder blocks. Therefore, they replace Decoder blocks
with similar Encoder blocks.

The most advantage of BERT is to apply two-dimensional training techniques of
Transformers from a very famous Attention model to a Language Model. Different
from previous NLP studies, which look at a text string from left to right,
BERT combines how to look at a text string from 2 dimensions (from left to
right and right to left). This method can greatly improve the retention of
Representations of words in sentences. The published results also show that
the trained language model has a more profound contextual meaning than
previous models.

Two sizes have been proposed for the BERT model:

  * •

$BERT_{BASE}$: 12 encoder block, 12 head attention, 110 million parameters.

  * •

$BERT_{LARGE}$: 24 encoder block, 16 head attention, 340 million parameters.

##  4 Methodology

For the sentiment analysis task of review texts, BERT can be used with two
approaches:

  * •

Feature extraction: This method uses BERT as a feature extraction model. The
architecture of the BERT model is preserved, and its outputs are feature input
vectors for subsequent classification models to solve the given problem.

  * •

Fine-tuning: In this method, we need to modify the architecture of the model
by adding some layers at the end of BERT model. These layers will solve the
problem, and retrain the model. This process is called fine-tuning. In [5],
this method is also the main method used to evaluate benchmarks on different
tasks, showing BERT superiority over previous models.

TABLE II: Comparison between Fine-tuning and Feature-based Approaches with
BERT [5]

SYSTEM | Dev F1 | Test F1
---|---|---
Fine-tuning approach |  |
$BERT_{LARGE}$ | 96.6 | 92.8
$BERT_{BASE}$ | 96.4 | 92.4
Feature-based approach ($BERT_{BASE}$) |  |
Embeddings | 91.0 | -
Second-to-Last Hidden | 95.6 | -
Last Hidden | 94.9 | -
Weighted Sum Last Four Hidden | 95.9 | -
Concat Last Four Hidden | 96.1 | -
Weighted Sum All 12 Layers | 95.5 | -

Table II (extracted from [5]) shows the performance of feature-based and fine-
tuning approaches with BERT on the Named Entity Recognition task(CoNLL-2003
NER [4]). Fine-tuning approaches achieve better performance than feature-based
approaches, where different combinations of hidden vectors are experimented
with. Therefore, in this paper, we decide to use the fine-tuning approach with
BERT for our sentiment analysis task of Vietnamese reviews.

To fine-tune BERT, we need to employ a pre-trained BERT model. It is required
that the pre-trained model has been trained with Vietnamese datasets. In this
work, we use the pre-trained BERT-Base Multilingual
Cased111https://github.com/google-research/bert provided by Google because it
supports multiple languages, including Vietnamese along with other languages
(104 languages). This pre-trained model is thus suitable for Vietnamese. Two
methods of fine-tuning BERT for sentiment analysis are as follows.

  * •

Fine-tuning BERT using token [CLS]: Devlin et al. [5] perform fine-tuning of
BERT at the sequence level. They add a special token to perform the
classification tasks. A token [CLS] is added to the beginning position of the
sentences. The output vector of this token will be sent through the
feedforward neutral network to perform input sentence classification (as shown
in Figure 1). This model is named BERT-base.

  * •

Fine-tuning BERT using all tokens: We use the entire output of BERT including
the token [CLS]. These outputs form a $SEQ\\_LEN\times h$ matrix, where
$SEQ\\_LEN$ is the maximum length of the input sequence, and $h$ is the length
of hidden vectors. We can then use this output matrix as the input to other
classification model which can be one of three models: LSTM, TextCNN, or RCNN
(as shown in Figure 2).

Figure 1: BERT-base architecture  Figure 2: BERT architecture using all tokens

Long Short-Term Memory (LSTM): We employ the Long Short-Term Memory (LSTM),
which is the most widely-used recurrent neural network (RNN) model, that
addresses the distance dependence issue of the classical RNN model. The LSTM
layer then continues to extract the features received from BERT.

Text Convolution Neural Network (TextCNN): The convolution model in this
research is TextCNN [18]. This is the CNN model widely used in nature language
processing tasks, especially for classification tasks thanks to the quick
training time and good results. The TextCNN model employed in this work is a
little different from the TextCNN models suggested in [18, 17]. We use 4
regions sizes (2,3,4,5).

Figure 3: TextCNN architecture [17]

Recurrent Convolutional Network (RCNN): The model is a combination of
recurrent and convolutional architectures [13]. Two LSTM architectures are
combined: one earns the context of words from left to right while the other
learns contextually from right to left. The output of both networks is then
passed through a conv1d layer to continue extracting. The RCNN architecture is
described in Figure 4.

Figure 4: RCNN Architecture

Figure 3, Figure 4 illustrate the architectures that will be associated with
BERT in this work. To perform classification of Positive and Negative labels,
we use the logistics function.

Based on the experimental results of [5], fine-tuning approaches give better
performance than feature-based approaches (as shown in Table II). Among the
feature-based approaches, the one that concatenate the last four hidden
outputs gives the best results. In this paper, we decide to also concatenate
the last four hidden outputs when performing fine-tuning. Each hidden output
has $dim=h$. Therefore, concatenating four hidden outputs results in the
dimensions of the output vector having $dim=4\times h.$

##  5 Experiments & Discussions

###  5.1 Datasets

We employ two datasets to train and evaluate all the models that are
investigated in this paper. The two datasets consist of reviewing texts
commented by users on e-commerce sites.

First, the Ntc-sv222https://streetcodevn.com/blog/sav dataset includes reviews
about food and restaurants on Foody. This dataset consists of 50,000 samples.
The labels are assigned based on the average score (avg_score): the ones above
8.5 are labeled positive, the ones less than 5 are labeled negative.

Second, the Vreview dataset is built on the training dataset of a competition
of AIVIVN333https://www.aivivn.com/contests/1 on sentiment analysis, which
consists users’ comments of product reviews on various e-commerce sites. In
addition, we include some comments about food and restaurants on
Foody444https://forum.machinelearningcoban.com/t/du-lieu-review-cua-foody/203.
We then perform data labeling through the average score (avg_score) similarly
to the Ntc-sv dataset. However, a slightly different thresholds are used here:
the samples with average scores above 7.5 are labeled positive, and the ones
below 5 are labeled negative.

Employing two datasets with different characteristics would help to better
evaluate the performance of different models. Some data statistics are made
for the evaluation process are showed in Tables III and IV. These are the
statistics after the data have been processed.

TABLE III: Data Description after Pre-process

Dataset | Train | Test | Totally
---|---|---|---
Positive | Negative | Positive | Negative
Ntc-sv | 20,493 | 20,267 | 5,000 | 5,000 | 50,760
Vreview | 22,979 | 19,537 | 8,301 | 6,795 | 57,612

TABLE IV: Statistics of words included in the comments

| Vreview | Ntc-sv
---|---|---
Mean | 55.45 | 86.57
Stdn | 63.75 | 77.41
Min | 1 | 1
25% | 14 | 37
50% | 32 | 65
75% | 76 | 111
Max | 435 | 1,501

###  5.2 Models

In this paper, we employ the following machine learning and deep learning
models in comparison with the BERT models for the sentiment analysis task over
our datasets of Vietnamese reviews.

  * •

SVM / Boosting: SVM and Boosting are two classic machine learning algorithms.
In this study, SVM experiments are based on $n$-grams features ($n$ in the
range [1,5]), while Boosting experiments are based on the XGBoosting algorithm
[15] (deep = 15).

  * •

FastText + LSTM / TextCNN / RCNN: We choose three deep learning architectures
to combine with the word embedding model FastText
555https://fasttext.cc/docs/en/crawl-vectors.html, which was pre-trained on a
Vietnamese dataset.

  * •

GloVE + LSTM / TextCNN / RCNN: Because we do not have pre-trained GloVE
embeddings with the same dimensions as the pre-trained FastText embedding, we
use the glove-python666https://github.com/maciejkula/glove-python library to
train the word embedding model. Models associated with GloVE would then be
similar to models combined with FastText, ensuring fair comparisons in our
experiments.

###  5.3 Results & Discussions

The performance results of all the competing models on the two datasets are
shown in Tables V and VI.

TABLE V: Experimental results on ntc-sv dataset

Model | Precision(%) | Recall(%) | F1(%)
---|---|---|---
SVM | 89.23 | 92.52 | 90.84
XGBoost | 88.76 | 90.58 | 89.63
FastText + TextCNN | 67.9 | 89.1 | 77.1
FastText + LSTM | 88.5 | 89.7 | 89.1
FastText + RCNN | 89.2 | 91.7 | 90.4
Glove + TextCNN | 69.7 | 87.7 | 77.7
Glove + LSTM | 88.7 | 91.8 | 89.8
Glove + RCNN | 85.8 | 85.8 | 90.7
BERT-base | 88.13 | 94.02 | 90.9
BERT-LSTM | 89.78 | 92.08 | 90.91
BERT-TextCNN | 88.85 | 93.14 | 90.94
BERT-RCNN | 88.76 | 93.68 | 91.15

TABLE VI: Experimental results on vreview dataset

Model | Precision(%) | Recall(%) | F1(%)
---|---|---|---
SVM | 86.26 | 86.9 | 86.5
XGBoost | 87.69 | 88.45 | 88.07
FastText + TextCNN | 61.8 | 94 | 74.6
FastText + LSTM | 88.5 | 86.4 | 87.5
FastText + RCNN | 84.5 | 89.8 | 87.1
Glove + TextCNN | 62.6 | 93 | 74.8
Glove + LSTM | 85.8 | 85.8 | 85.8
Glove + RCNN | 84.0 | 88.6 | 86.2
BERT-base | 86.08 | 88.44 | 87.2
BERT-LSTM | 85.25 | 89.9 | 87.5
BERT-TextCNN | 90.9 | 85.2 | 87.98
BERT-RCNN | 87.08 | 89.38 | 88.22

The following conclusions can be drawn from the experiments with BERT-based
approaches and other approaches:

  * •

When comparing the models on the ntc-sv dataset, it can be seen that the
results are not considerably different, even when comparing BERT models with
SVM. Therefore, deep learning models are not much superior in this case. The
training time of BERT-based approaches is much longer than the training time
of the SVM approach; however, their performances are not much better than
SVM’s.

  * •

Models using the word embeddings FastText and GloVe produce similar results.
First, TextCNN models produce very poor results, much lower than other models.
Secondly, the models using architectures of LSTM and RCNN, especially models
using RCNN architecture produce very good results, because it combines the
features that are received from many sources, from which there is more
information available for better classifying the reviews. Their training
times, however, are considerably longer.

  * •

The BERT-based, BERT-LSTM, and BERT-TextCNN approaches similarly exhibit good
performance on both datasets.

  * •

BERT-RCNN is the model that gives the best results among the models considered
in this paper. It can be seen that RCNN is not only suitable when combined
with BERT but also effective in combination with other embedding methods. The
experimental results here suggest that RCNN is the most suitable approach to
be combined with BERT to implement fine-tuning.

The experimental results did not show a considerable difference between BERT
and other models, although we used a different fine-tuning approach, which
made the most of the information from BERT. These results can be explained
with the following reasons:

  * •

The lengths of the comments are relatively short and the pre-processing
procedure eliminates most of the potential confusions, making it easier to
classify the comments.

  * •

All the fine-tuning BERT approaches, BERT-base and BERT in combination with
other deep learning architecture, require a lot of computational resources,
namely RAM and GPU. The input string for BERT after sub-splitting is 512, but
our resources are not enough to be able to fine-tune BERT with the input
string length of 512. If we had enough resources to process the maximum input
length, the performance of BERT-based methods could be further improved.

  * •

Pre-trained BERT-Base Multilingual Cased might still be sub-optimal for
Vietnamese language. The sub-word separator in BERT uses the sub-word
separation technique for English, which might lead to inaccurate separations
when being applied to Vietnamese. The pre-trained BERT model, therefore, has
certain contextual deviations for Vietnamese words.

The source code, models and datasets of this study can be downloaded at:
https://github.com/thoailinh/Sentiment-Analysis-using-BERT.

##  6 Conclusion

We have fine-tuned BERT using the pre-trained multilingual BERT with two
approaches and both exhibit better performance compared to other machine
learning and deep learning models considered in this paper. When performing
fine-tuning BERT, the entire output of BERT can be employed, providing the
subsequent classification model with more useful information. The experimental
results showed that fine-tuning BERT with LSTM or TextCNN yielded no
considerable improvement compared to the BERT-base fine-tuning approach.
Instead, BERT could be integrated with models using RCNN or other
architectures that combine recurrent and convolutional models.

The experimental results show that the BERT-RCNN model using our proposed
fine-tuning method yields certain improvement of the accuracy performance for
sentiment analysis on datasets containing Vietnamese reviews. Although the
improvement is not considerably superior, it exhibits potentials for further
enhancements. For future work, we aim to extend the proposed method for
aspect-based sentiment analysis, and more extensive experiments could be
conducted on different datasets.

## References

  * [1] Lopamudra Dey Sanjay Chakraborty et al. , “Sentiment Analysis of Review Datasets Using Naïve Bayes and K-NN Classifier”, in International Journal of Information Engineering and Electronic Business , 2016.
  * [2] Ashish Vaswani et al. “Attention is all you need”, 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.
  * [3] Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan, “Sentiment classification using machine learning techniques”, in Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP-2002), 2002.
  * [4] Erik F Tjong Kim Sang and Fien De Meulder, “Introduction to the conll-2003 shared task: Language-independent named entity recognition”. In CoNLL (2003).
  * [5] Jacob Devlin, et al. “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”. 2018.
  * [6] Bhumika M. Jadav and Vimalkumar B. Vaghela, “Sentiment Analysis using Support Vector Machine based on Feature Selection and Semantic Analysis”, in International Journal of Computer Applications 2016.
  * [7] Nurulhuda Zainuddin and Ali Selamat, “Sentiment analysis using Support Vector Machine”, 2014 International Conference on Computer, Communications, and Control Technology (I4CT).
  * [8] Jeffrey Pennington, Richard Socher, Christopher Manning, “GloVe: Global Vectors for Word Representation”, in Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014.
  * [9] Maite Taboada, Julian Brooke, Milan Tofiloski, Kimberly Voll, and Manfred Stede, “Lexicon-Based Methods for Sentiment Analysis”, Computational Linguistics, vol. 37, no. 2, pp. 267-307, 2011.
  * [10] Khuong Vo, Tri Nguyen, Dang Pham, Mao Nguyen, Minh Truong, Dinh Nguyen, Tho Quan, “Handling negative mentions on social media channels using deep learning”, Journal of Information and Telecommunication, vol. 3, no. 3, 2019.
  * [11] Nguyen Thi Duyen, Ngo Xuan Bach, Tu Minh Phuong, “An Empirical Study on Sentiment Analysis for Vietnamese”. The 2014 International Conference on Advanced Technologies for Communications (ATC’14).
  * [12] Peter D. Turney, “Thumbs up or thumbs down?: semantic orientation applied to unsupervised classification of reviews”, in Proceedings of Annual Meeting of the Association for Computational Linguistics (ACL- 2002), 2002.
  * [13] Siuwei Lai, Liheng Xu, Kang Liu, Jun Zhao, “Recurrent Convolutional Neural Networks for Text Classification”, Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence (2015).
  * [14] Kavya Suppala, Narasinga Rao, “Sentiment Analysis Using Naïve Bayes Classifier”, in International Journal of Innovative Technology and Exploring Engineering (IJITEE) 2019
  * [15] Tianqi Chen, Carlos Guestrin, “XGBoost: A Scalable Tree Boosting System”, The 22nd ACM SIGKDD International Conference (2016).
  * [16] Xiang Zhang, Junbo Zhao, Yann LeCun, “Character-level Convolutional Networks for Text Classification”, arXiv:1509.01626, 2015.
  * [17] Ye Zhang, Byron C. Wallace, “A Sensitivity Analysis of (and Practitioners’ Guide to) Convolutional Neural Networks for Sentence Classification”, IJCNLP 2017.
  * [18] Yoon Kim “Convolution Neutral Network for Sententce Classification”, Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP).
  * [19] Quoc Le, Tomas Mikolov, “Distributed Representations of Sentences and Documents”, in Proceedings of the International Conference on Machine Learning (ICML 2014), 2014\.
  * [20] Quan-Hoang Vo, Huy-Tien Nguyen, Bac Le, Minh-Le Nguyen, “Multi-channel LSTM-CNN model for Vietnamese sentiment analysis”, 2017 9th International Conference on Knowledge and Systems Engineering (KSE).
  * [21] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean , “Distributed Representations of Words and Phrases and their Compositionality”, in Proceedings of the 26th International Conference on Neural Information Processing Systems (NIPS 2013), 2013.