# Human Activity Recognition using Attribute-Based Neural Networks and Context
Information

Stefan Lüdtke1111These authors contributed equally to the paper. Fernando Moya
Rueda2∗ Waqas Ahmed1 Gernot A. Fink2&Thomas Kirste1 1Institute of Visual &
Analytic Computing, University of Rostock, Germany
2Department of Computer Science, TU Dortmund University, Germany
{stefan.luedtke2, waqas.ahmed<EMAIL_ADDRESS>
{fernando.moya<EMAIL_ADDRESS>
###### Abstract

We consider human activity recognition (HAR) from wearable sensor data in
manual-work processes, like warehouse order-picking. Such structured domains
can often be partitioned into distinct _process steps_ , e.g., packaging or
transporting. Each process step can have a different prior distribution over
_activity classes_ , e.g., standing or walking, and different system dynamics.

Here, we show how such context information can be integrated systematically
into a deep neural network-based HAR system. Specifically, we propose a hybrid
architecture that combines a deep neural network—that estimates high-level
movement descriptors, _attributes_ , from the raw-sensor data—and a shallow
classifier, which predicts activity classes from the estimated attributes and
(optional) context information, like the currently executed process step.

We empirically show that our proposed architecture increases HAR performance,
compared to state-of-the-art methods. Additionally, we show that HAR
performance can be further increased when information about process steps is
incorporated, even when that information is only partially correct.

##  1 Introduction

The accurate recognition of human activities from sensor data is an important
task for many applications, like healthcare, sports, or for developing
situation-aware assistive technologies. As another example, activity
recognition is relevant for the analysis and optimization of manual-work
processes, like packaging in a warehouse, see Fig. 1, the main motivation for
our work Reining et al. (2018b); Niemann et al. (2020b).

Human Activity Recognition (HAR) from movement data is challenging due to the
intra- and inter-subject variability of human movement. Recently, deep neural
networks have been used successfully for multichannel time-series HAR. These
networks combine the feature extraction and classification in an end-to-end
approach Ordóńez and Roggen (2016); Zeng et al. (2014); Ronao and Cho (2015);
Hammerla et al. (2016); Yao et al. (2018). They learn non-linear and temporal
relations of basic, complex, and highly dynamic human movements directly from
raw-inertial sensor data. These transformations are more discriminative with
respect to human action classes than hand-crafted features Zeng et al. (2014);
Hammerla et al. (2016).

However, such deep neural networks are usually purely data-driven—they cannot
directly make use of prior knowledge that is often available in highly
structured application domains. For example, manual work processes, like order
picking or packaging, are structured into distinct high-level _process steps_
, as shown in Fig. 4. Each of these steps has a unique prior distribution over
activity classes and unique system dynamics. For example, a person is more
likely to perform a _handling_ activity class when retrieving articles in the
warehouse (step 3 in Fig. 4) than when moving the cart (step 2 in Fig. 4).
Furthermore, information about the currently executed process step is often
available directly from external sources. For example, in a packaging
scenario, each article and empty box are scanned before being packed,
according to a shipment list. This scanning process, which is recognized in
the warehouse management system, indicates a transition between process steps.

Figure 1: Physical laboratory set-up of a warehouse packaging work station.

In this paper, we show how such additional _context information_ can be
integrated systematically into a deep neural network-based HAR system.
Specifically, we propose a hybrid architecture that allows integrating context
information without re-training the neural network from scratch: We use a deep
neural network that extracts high-level movement descriptors (_attributes_ ,
like posture or usage of left or right hand) from raw wearable-sensor data. On
top, a shallow classifier estimates activity classes from the attributes and
optionally from the context information—like the currently executed process
step, in our case. This way, we can use a deep neural network independently of
available context information and domain specifics. Instead, we only need to
adapt the final classifier to different domains, i.e., different process
models or changes in the available context information.

We evaluate this architecture on a HAR task in intra-logistics, and
empirically show that it achieves increased HAR performance, compared to the
state of the art, even when no context information is available. Additionally,
the HAR performance increases further when context information in the form of
the currently executed process step is available—even when it is just
partially correct.

##  2 Related Work

Methods of statistical pattern recognition are common for analyzing human
movements from measurements from on-body devices. A standard pipeline involves
pre-processing, segmentation, hand-crafted feature extraction, and
classification. Recently, deep learning methods become the standard method for
solving HAR problems in gesture recognition and activities of daily living
Ordóńez and Roggen (2016); Grzeszick et al. (2017); Hammerla et al. (2016);
Ronao and Cho (2015); Yao et al. (2018). Deep architectures combine the
feature extraction and classification in an end-to-end approach. These
architectures learn the non-linear and temporal relations of basic, complex,
and highly dynamic human movements. They learn non-linear transformations
directly from raw-inertial data. Compared to the hand-crafted ones, these
transformations are more discriminative to human actions and invariant to
distortions and temporal translations Hammerla et al. (2016). The authors in
Ronao and Cho (2015) introduced temporal convolutional neural networks
(tCNNs), which carry out convolution and pooling operations along the time
axis. These architectures share rather small convolution filters among all the
sensors, as local temporal measurements are correlated independently of the
sensor type; this is also valid assuming a normalization per sensor. An
architecture that combines temporal convolutions and recurrent networks is
introduced in Ordóńez and Roggen (2016). Specifically, the authors used Long
Short-Term Memory units (LSTMs), recurrent units with memory cells, and a
gating system Hochreiter and Schmidhuber (1997). The authors in Hammerla et
al. (2016) utilized shallow recurrent-networks: a three-layered LSTM and a
one-layered bidirectional LSTM. The authors in Grzeszick et al. (2017) created
a tCNN that adapts to the wearables per human limb. This architecture contains
parallel-convolutional branches per human limb. Each convolutional branch
creates a deep representation of the measurements per human limb. The network
deploys a late fusion for creating a final deep representation. The authors in
Qian et al. (2019) proposed an end-to-end architecture that extract temporal,
body-relations and statistical features.

Nonetheless, the performance of these deep learning methods did not show a
significant increase compared to other areas such as image and video
classification or segmentation Hammerla et al. (2016); Grzeszick et al.
(2017). HAR remains a challenging task due to large intra- and inter-class
variability of human movements, i.e., humans carry out similar tasks
differently. In addition, there is a broad range of human activities or
movements, and there is not a standard definition nor structure for
formulating a clear problem of HAR Bulling et al. (2014). Likewise, datasets
for HAR suffer from the class-unbalance problem, where the number of samples
per action class differs strongly Ordóńez and Roggen (2016).

Attribute representations help solve zero-shot learning and transfer learning.
High-level attribute representations are semantic descriptions that describe
categories, e.g., in object and scene recognition problems Cheng et al.
(2013); Lampert et al. (2014); Zheng et al. (2017) and words in document
analysis. In object recognition, attributes can be the shape, color, texture,
size of objects, or even geographic information. Different approaches have
introduced semantic descriptions of activities as representations for solving
HAR Cheng et al. (2013); Arif Ul Alam and Roy (2017); Zheng et al. (2017). In
Cheng et al. (2013), the authors proposed to use semantic attributes for
recognizing unseen activities. In Zheng et al. (2017), human-annotated
attributes and data-driven attributes are combined for solving HAR in sports
videos. They selected a subset from both attribute groups maximizing the
discrimination capability of attributes for distinguishing different sets of
classes. The authors in Arif Ul Alam and Roy (2017) designed a hierarchical
representation of human-activity taxonomy based on semantic descriptions in
the context of smart-home applications.

Precisely, attribute-based representations have been deeply explored on HAR in
the manual order picking process in Reining et al. (2018b). Attribute
representations are beneficial for dealing with the versatility of activities.
The authors in Reining et al. (2018b) compared the performance of deep
architectures trained using different attribute representations, evaluating
their quantitative performance and quality from the perspective of practical
application. The authors in Reining et al. (2018c) tested different attribute
representations, expert-given and random, for solving zero-shot learning in
HAR. Unseen activities were described using attributes that are shared with
the seen activities. Expert-given attribute representations performed better
than a random one. The latter is created following the conclusions in Moya
Rueda and Fink (2018). A semantic relation between attributes and activities
enhances HAR not only quantitatively with regards to performance but also
guarantees a transfer of the attributes between activities by domain experts.
In this preliminary work, the mapping between activity classes and attribute
representations was one-to-one. Fig. 2 presents an example of attribute-based
representation for HAR.

Deep neural networks do not consider domain knowledge about the causal
relation of activities that is often available in highly structured domains.
To overcome this limitation, a combination of symbolic reasoning methods
(Computational State-Space Models, CSSM Krüger et al. (2014)) and deep neural
networks is introduced in Moya Rueda et al. (2019). The CSSM models prior
knowledge about the high-level, causal structure of the domain, and the deep
neural network acts as observation model, relating the sensor data to CSSM
states.

##  3 Methods

Steps…Transporting Box…Activities… Standing Walking Cart Handling Synchr.
…AttributesGait CycleTorso Rot.Right HandLeft HandHandy UnitNo itemDataTime
[s] Figure 2: Relationship between attributes, process steps and activity
classes. First, attributes are estimated from the raw sensor data. Activities
are then classified based on these attributes and the current process step.

We propose a hybrid HAR method, consisting of a deep neural network that
predicts movement descriptors (attributes) from the sensor data, and a model
that predicts activity classes from the attribute estimates.

In the following, we first present the architecture of the neural network in
more detail. Afterwards, we discuss different options for representing and
learning the relationship between attributes and activity classes. Finally, we
show how context information can be integrated into the classification system.

###  3.1 Attribute-based Deep Neural Network for HAR

We deploy the temporal convolutional neural network (tCNN) from Yang et al.
(2015); Grzeszick et al. (2017). It has shown to perform relatively well
despite its simplicity Grzeszick et al. (2017); Moya Rueda and Fink (2018). A
tCNN is an end-to-end architecture composed of feature extractors and a
classifier, either a softmax or a sigmoid. The architecture processes
sequences of size $[T,W]$, with $T$ the sequence length and $W$ the number of
sequence channels. The tCNN contains four convolutional layers, no
downsampling, and two fully connected layers, and a classifier. The
convolutional layers are composed of $64$ filters of size $[5\times 1]$,
performing convolutions along the time axis. The first and second fully
connected layers contain $128$ units. Depending on the task, the tCNN will
have a softmax layer for activity classification or a sigmoid layer for
attribute classification Moya Rueda and Fink (2018). Here, a sigmoid
activation function replaces the usual softmax layer, as we use the
tCNNattribute to compute an attribute representation from an input sequence,
rather than directly classifying it, following Moya Rueda and Fink (2018).
Using attribute representations have shown to be beneficial for HAR, following
the conclusions in Moya Rueda and Fink (2018); Niemann et al. (2020b). The
tCNNattribute is trained using the binary-cross entropy loss. Fig. 2 presents
the relation between activities, attributes, and sequence input, and Fig. 3
shows the tCNNattribute architecture.

Due to the final sigmoid layer, the tCNNattribute output can be interpreted as
the probabilities for each attribute being present or not present in the input
segment. More formally, the tCNNattribute is a function
$\phi:D\rightarrow\Pi$, where $d\in D$ is a data sample, i.e., segment, and
$\pi\in\Pi$ is a parameter vector for the posterior distribution over
attributes $p_{\phi}(a\operatorname{|}d)$. Specifically, the probability of a
binary attribute vector $a=(a_{1},\dots,a_{K})$ is given by a product of
Bernoulli distributions:

$\displaystyle
p_{\phi}(a\operatorname{|}d)=\prod_{k=1}^{K}p(a_{k}\operatorname{|}d)=\prod_{k=1}^{K}\pi_{k}^{a_{k}}\,(1-\pi_{k})^{1-a_{k}}$
(1)

Sensor DatatCNNSigmoidApproximations of
$p_{\phi}(a\operatorname{|}d)$$\pi_{1},\pi_{2},...,\pi_{K}$Context
Information$S$Shallow Classifiers(QDA, HMM, RF)Activity Class
Estimate$\hat{C}$ Figure 3: Proposed hybrid activity recognition architecture.
The neural network provides probabilities $\pi_{i}$ that each of the
attributes $a_{i}$ is present. The probability vector $\pi$ and the context
information (e.g. the current process step) is used as input of a shallow
classifier that estimates an activity class $\hat{c}$.

###  3.2 Human Activity Recognition from Semantic Attributes

From an application viewpoint, we are interested in the activity class $c$
instead of the attributes themselves. In general, the maximum-likelihood
estimate $\hat{c}$ of the activity class is given by

$\hat{c}=\text{argmax}_{c}\,p(c\operatorname{|}d)$ (2)

We assume that the activity class $c$ and the sensor data $d$ are
conditionally independent, given the attribute vector $a$, so that the
activity class posterior can be written as

$\displaystyle p(c\operatorname{|}d)=\sum_{a\in
A}p(c\operatorname{|}a)\,p_{\phi}(a\operatorname{|}d).$ (3)

Here, $p_{\phi}(a\operatorname{|}d)$ is given directly by the tCNN $\phi$, as
shown in Section 3.1 (different options for modeling $p(c\operatorname{|}a)$
are discussed below). Thus, in principle, an estimate of the activity class
$\hat{c}$ for a given data segment $d$ is obtained in two steps:

  1. (i)

Compute $\pi$ via the neural network $\phi$: $\pi=\phi(d)$

  2. (ii)

Compute Eq. 2 with these values $\pi$

In the following, we present two options for modeling $p(c\operatorname{|}a)$:
The _Direct Attribute Prediction_ (DAP), proposed by Lampert et al. (2014),
and the _Generalized DAP_ , that we formally introduce here for the first
time, but that was implicitly used by Moya Rueda and Fink (2018); Niemann et
al. (2020b) before in an approximate form. Finally, we describe a novel idea
that does not explicitly use the probabilistic model above, but uses a
classifier to model the dependency between network outputs $\pi$ and activity
classes $c$.

#### Direct Attribute Prediction

The _Direct Attribute Prediction_ (DAP) model Lampert et al. (2014) assumes
that each class $c$ has an associated unique attribute representation
$a^{(c)}$, i.e. there is a deterministic relationship $f(c)=a^{(c)}$ so that
$p(a\operatorname{|}c)=\mathds{1}(a=a^{(c)})$. By making use of Bayes’
theorem, one can rewrite
$p(c\operatorname{|}a)=\frac{p(c)}{p(a)}p(a\operatorname{|}c)$. Inserting back
into Eq. 3 gives

$\displaystyle p(c\operatorname{|}d)$ $\displaystyle=\sum_{a\in
A}\frac{p(c)}{p(a)}p(a\operatorname{|}c)\,p_{\phi}(a\operatorname{|}d)$ (4)
The normalization factor $p(a)$ is constant w.r.t. the class. Furthermore,
Lampert et al. (2014) propose to use a uniform class prior $p(c)$. Thus, the
factor $\frac{p(c)}{p(a)}$ can be ignored for classification, so that the
expression simplifies to $\displaystyle p(c\operatorname{|}d)$
$\displaystyle\propto\sum_{a\in
A}\mathds{1}(a=a^{(c)})\,p_{\phi}(a\operatorname{|}d)$ (5)
$\displaystyle=p_{\phi}(a^{(c)}\operatorname{|}d),$ (6)

where $a^{(c)}=f(c)$ is the unique attribute representation of class $c$.

#### Generalized Direct Attribute Prediction

Unfortunately, in HAR, activity classes usually do not have a unique attribute
representation. For example, the activity class _take_ can involve the
attribute _left hand_ and/or _right hand_. Thus, the DAP model cannot be used
directly.

However, a deterministic relationship in the opposite direction often exists
in HAR, i.e., each attribute vector corresponds to exactly one activity class.
This mapping $g(a)=c$ can either be defined in advance from prior domain
knowledge, or estimated from training data.

In either way, the distribution becomes
$p(c\operatorname{|}a)=\mathds{1}(c=g(a))$, which can be inserted into Eq. 2
and 3. In this case, the maximum-likelihood estimate is computed as

$\hat{c}=\text{argmax}_{c}\sum_{\\{a\in
A\,|\,g(a)=c\\}}p_{\phi}(a\operatorname{|}d)$ (7)

We call this model _Generalized_ Direct Attribute Prediction (GDAP).

However, computing the sum in Eq. 7 exactly is only feasible as long as the
number of attribute vectors $|A|$ is small. To make classification feasible
even when $|A|$ is large, Moya Rueda and Fink (2018); Niemann et al. (2020b)
propose to compute the sum not exactly, but to compute the maximum-likelihood
approximation222More precisely, Moya Rueda and Fink (2018); Niemann et al.
(2020b) do not compute
$\hat{a}=\text{argmax}_{a}\,p_{\phi}(a\operatorname{|}d)$, but
$\hat{a}=\text{argmin}_{a}\,||a-\pi||_{2}$ after normalizing $a$ and $\pi$,
hence we call this model _nearest neighbor approximation_.
$\hat{a}=\text{argmax}_{a}\,p_{\phi}(a\operatorname{|}d)$, and return
$\hat{c}=g(\hat{a})$. That is, their approximation assumes that the sum in Eq.
7 is dominated by its largest term, similar to the approximation done in
Ramírez and Geffner (2010) for goal recognition.

#### Classification-based methods

The former methods either require to sum over all attribute vectors (which
quickly becomes infeasible, as the number of possible attribute vectors grows
exponentially with the number $K$ of attributes), or need to make strong
assumptions that might induce a large error.

Therefore, we propose a novel approach to obtain activity classes, shown in
Fig. 3. The key insight is to view the neural network outputs
$\pi_{1},\dots,\pi_{n}$ not as parameters of $p_{\phi}(a\operatorname{|}d)$,
but as high-level _features_ or _latent representations_ that can be used as
input of a classifier that learns the function $c=h(\pi)$.

In this case, an estimate of the activity class $\hat{c}$ is for a given data
segment $d$ obtained in two steps: (i) Compute the neural network output
$\pi=\phi(d)$; (ii) Compute the activity class estimate $\hat{c}=h(\pi)$.
Specifically, we use Quadratic Discriminant Analysis (QDA), a Hidden Markov
Model with Gaussian observation model (HMM) and Random Forests (RF) as
classifiers.

In addition to the potentially higher computational efficiency during
classification, this concept has two key advantages over the probabilistic
models introduced before:

  * •

It allows to account for bias in the neural network, i.e., the shallow
classifier can produce a correct classification even when the network output
$\pi$ does not assign a high probability to attribute vectors associated with
the true class.

  * •

Integrating process step information as an additional feature of the
classifier becomes straightforward, and does not require re-training of the
complete neural network when additional process information becomes available,
as discussed next.

###  3.3 Making use of Process Knowledge

Figure 4: The Business Process Model (BPM) for the logistics scenario.

In highly structured domains like manual-work processes, the overall activity
sequence can often be separated into distinct _process steps_ , see Fig. 2. As
a concrete example, consider the Business Process Model (BPM) in Figure 4 that
describes a simple order-picking process in a warehouse. At each point in
time, the subject is performing exactly one of these process steps.

Here, we assume that an estimate of the current process step is available. For
example, in the warehouse scenario, the worker needs to scan each article
before putting it inside a box for further transportation or packing. These
scanning events, thus, allow recognizing process-step transitions accurately.
In other domains, information about process transitions might not be available
directly, but can be estimated by causal models, like Computational State-
Space Models Krüger et al. (2014).

In principle, such process step information could be used directly as
additional input to the attribute network $\phi$. However, this would require
re-training of the network, which can be computationally very costly.
Additionally, the original training data still need to be available to allow
re-training, which might not be the case when using a pre-trained network as a
black-box model. Furthermore, we assume that the data-attribute dependency
that is modeled by the neural network is independent of the process step—how
the sensor observations relate to poses and movements of the subject, i.e.,
the attributes, does not change when the subject is executing different sub-
tasks.

BPM state |  Stand |  Walk |  Cart |  Handle (up) |  Handle (center) |  Handle (down) |  Sync.
---|---|---|---|---|---|---|---
S0 |  0.266 |  0.375 |  0.000 |  0.047 |  0.195 |  0.031 |  0.086
S1 |  0.000 |  0.053 |  0.000 |  0.342 |  0.474 |  0.132 |  0.000
S2 |  0.011 |  0.000 |  0.757 |  0.006 |  0.215 |  0.011 |  0.000
S3 |  0.061 |  0.000 |  0.000 |  0.239 |  0.534 |  0.166 |  0.000
S4 |  0.000 |  0.095 |  0.000 |  0.167 |  0.500 |  0.238 |  0.000
Table 1: Distribution of activity classes for each BPM state.

Instead, we assume that only the relationship between _attributes and activity
classes_ depends on the process step. Therefore, we propose to include the
process step information into the shallow classifier that predicts activity
classes $c$ from network outputs $\pi$. More specifically, we make use of the
process knowledge in the different classifiers in the following ways:

  * •

In the Random Forest, we add the process step as an additional input feature.

  * •

For the QDA, we estimate different priors $p_{s}(c)$ for each process step
$s$, but use a single, shared Gaussian likelihood $p(\pi\operatorname{|}c)$,
because the class priors differ substantially in each process step (see Table
1), but the amount of data is insufficient to estimate the parameters of
individual likelihoods for each process step.

  * •

For the HMM, we train different transition models
$p_{s}(c_{t}\operatorname{|}c_{t-1})$ for each process step $s$, to account
for the different system dynamics of the different process steps. Similar to
the QDA above, the model uses a shared Gaussian observation likelihood
$p(\pi\operatorname{|}c)$.

##  4 Experimental Evaluation

###  4.1 Dataset: LARa

The Logistic Activity Recognition Challenge (LARa) dataset Niemann et al.
(2020b, a) is used to evaluate our approach. The dataset recreates three
warehousing scenarios in a constrained environment, ensuring natural motion
and resemblance to reality Reining et al. (2018a). This dataset contains
measurements of a marker-based MoCap system, called LARa-MoCap, and on-body
devices, called LARa-OB, from $14$ humans performing activities in the intra-
logistics. LARa provides recordings from the 3D joint-poses and the 3D linear
and angular acceleration of subjects performing eight activities. Joint poses
are recorded with a rate of $200$ Hz, and linear and angular accelerations
with $100$ Hz. The activities are common activities in the intra-logistics:
Standing, Walking, Moving Cart, Handling (upwards), Handling (centred),
Handling (downwards), Synchronization and None. Here, we experimented with
LARa-MoCap. For LARa-MoCap, there are $22$ joints. For each joint, LARa
provides the $3D$ pose. These are all centered with respect to the lower back
of a subject. In general, LARa provides $714$ min of annotated recordings
being a large annotated dataset for HAR.

The LARa dataset consists of of non-overlapping training, validation, and
testing sets. For LARa-MoCap, the validation and testing sets contain
recordings from subjects $[5,11,12]$. The training set contains recordings
from the other eight subjects of LARa-MoCap. Our experiments have used the
test set, which contains annotations of process steps from one of the three
warehouse scenarios.

###  4.2 Results

| No Process Info | With Process Info
---|---|---
Classifier | Acc. | F1 | Acc. | F1
GDAP | 0.636 | 0.628 | – | –
NN | 0.621 | 0.614 | – | –
QDA | 0.670 | 0.650 | 0.721 | 0.711
HMM | 0.685 | 0.672 | 0.719 | 0.708
RF | 0.688 | 0.674 | 0.710 | 0.715
Table 2: Experimental results, best results are printed in boldface. For each
model, we compared a baseline case where no process information is available
and the case where the correct BPM state at each time step. For GDAP and NN,
information about process states cannot be integrated directly. GDAP:
Generalized Direct Attribute Prediction; NN: Nearest neighbor-approximation of
GDAP; QDA: Quadratic Discriminant Analysis; HMM: Hidden Markov Model; RF:
Random Forest.

We compared the different methods, introduced in Section 3.2), for obtaining
activity classes from the tCNNattribute output. Specifically, we compared our
GDAP model, the nearest-neighbor (NN) approximation of GDAP proposed by
Niemann et al. (2020b), and three classification-based methods (QDA, HMM and
RF). All experiments were performed in R R Core Team (2020), and we used the
randomForest Liaw and Wiener (2002) with $500$ trees for fitting RFs.

We used a pre-trained tCNN for attribute prediction for all experiments, with
the architecture described in Section 3.1. The pre-trained tCNN can be found
in Niemann et al. (2020a). The network was trained on the full LARA-MoCap
dataset, as described in Niemann et al. (2020b). Furthermore, for the
classification-based methods, we investigated the case where the correct BPM
state (see Fig. 4) is provided to the models for each time step, see Section
3.3 for how this feature is utilized in the different classifiers.

Table 2 shows the HAR performance of the different methods. When the currently
performed BPM state is not provided to the models, the classification-based
methods (QDA, HMM, RF) all outperform the probabilistic models (GDAP, NN). Of
the classification-based methods, the RF achieves highest accuracy (0.668) and
F1 score (0.674) – higher than prior results by Niemann et al. (2020b) (NN,
with 0.621 accuracy and 0.621 F1 score).

To assess the information contained in the BPM state, we first computed the
mutual information $I(C,S)=H(C)-H(C\operatorname{|}S)$ of activity classes $c$
and BPM states $s$ (where $H(C)$ the marginal entropy of activity classes and
$H(C\operatorname{|}S)$ is the conditional entropy): On the test data, we
obtain $H(S)=2.44\,\text{bit}$ and $H(C\operatorname{|}S)=1.55\,\text{bit}$.
Thus, knowledge of the BPM states provides $I(C,S)=0.89\,\text{bit}$ of
additional information for classification. This finding is reflected in the
classification performance: The performance of all models increases when they
are provided with the correct BPM state at each timestep (see Table 2). Again,
all three classifiers show similar performance, with RF having the highest F1
score (0.715), and QDA having the highest accuracy (0.721).

Figure 5: F1 score w.r.t. accuracy of BPM state prediction. Horizontal lines
indicate the baseline performance without BPM state information. Each
experiment has been repeated 10 times, the ribbons indicate standard deviation
of the F1 score. The RF is more robust regarding inaccurate state information
than the QDA.

Next, we investigated how inaccuracies in the provided BPM state influence HAR
performance. Specifically, we introduced inaccuracies into the BPM state
feature as follows: With probability $p$, each BPM state $s_{t}$ is replaced
by a different state, chosen uniformly among the remaining states $S\setminus
s_{t}$. Thus, the noisy state sequence has an accuracy of $1-p$. We varied the
parameter $p$ between $0$, i.e., perfectly accurate state information, and
$0.5$, in steps of $0.05$, and investigated how this affected the F1 score of
the HAR models.

Fig. 5 shows the results for this experiment. For both the QDA and the RF, the
F1 score decreases when the BPM state estimate becomes less accurate. However,
the RF is more robust in that regard: Even when the accuracy of the BPM state
estimate is only 50%, the RF is not worse than the baseline RF that does not
make use of the BPM state at all – in contrast to the QDA, which requires a
state accuracy of at least 80% to outperform the QDA baseline.

In summary, the results indicate that providing the shallow classifiers with
context information in the form of the currently performed BPM state increases
the overall HAR performance, even when the BPM state is not always accurate.

##  5 Discussion and Conclusion

In this paper, we proposed a HAR system that combines a deep neural network
and a shallow classifier. The system predicts activity classes from the neural
network output, i.e., from the approximations of the posterior distribution
over attributes. Our empirical evaluation on a HAR task in logistics shows
that this combined method can increase HAR performance, compared to previous
methods Niemann et al. (2020b). Furthermore, this system allows integrating
additional context information (like an estimate of the current process step,
as used here) directly, without re-training the neural network. We showed that
providing an estimate of the current process step as an additional feature can
increase HAR performance, even when the process-step estimation is not always
correct.

However, these results are preliminary: Our evaluation is based on only a
subset of the complete LARa dataset, as process step annotations were only
available for one of the three scenarios contained in the dataset.
Furthermore, we just simulated the noise on the process step estimate. Future
work needs to confirm our findings for a more realistic case where the process
step is estimated from external sources, as well as on additional HAR
datasets. On a technical level, future work will investigate using an
additional neural network layer for predicting activity classes from
attributes instead of using a shallow classifier, thus allowing end-to-end-
training of the complete HAR system.

Depending on the sensor setup and the specific domain, additional context
information, other than the current process step, might be available, e.g.,
currently handled objects or the location of the subjects. Our proposed system
architecture allows using such features directly in a systematic way.
Investigating such cases is another topic for future work.

Finally, the BPM can be seen as a form of prior domain knowledge that we aim
to exploit. So far, we only made use of the fact that different BPM states
have different prior probabilities of activities, but the BPM also provides
information about the _dynamics_ of states and action. In the future, we will
explore how such information can serve to generate symbolic precondition-
effects models of the system dynamics, e.g., Computational State-Space Models
Krüger et al. (2014).

## References

  * Arif Ul Alam and Roy [2017] Mohammad Arif Ul Alam and Nirmalya Roy.  Unseen Activity Recognitions: A Hierarchical Active Transfer Learning Approach.  pages 436–446. IEEE, 2017.
  * Bulling et al. [2014] Andreas Bulling, Ulf Blanke, and Bernt Schiele.  A Tutorial on Human Activity Recognition Using Body-worn Inertial Sensors.  ACM Comput. Surv., 46(3):33:1–33:33, January 2014.
  * Cheng et al. [2013] Heng-Tze Cheng, Martin Griss, Paul Davis, Jianguo Li, and Di You.  Towards zero-shot learning for human activity recognition using semantic attribute sequence model.  In Proc. of the 2013 ACM Int. Joint Conf. on Pervasive and Ubiquitous Computing, UbiComp ’13, page 355–358, New York, NY, USA, 2013. Association for Computing Machinery.
  * Grzeszick et al. [2017] Rene Grzeszick, Jan Marius Lenk, Fernando Moya Rueda, Gernot A Fink, Sascha Feldhorst, and Michael ten Hompel.  Deep neural network based human activity recognition for the order picking process.  In iWOAR. ACM, 2017.
  * Hammerla et al. [2016] Nils Y. Hammerla, Shane Halloran, and Thomas Ploetz.  Deep, Convolutional, and Recurrent Models for Human Activity Recognition using Wearables.  arXiv:1604.08880 [cs, stat], April 2016.  arXiv: 1604.08880.
  * Hochreiter and Schmidhuber [1997] Sepp Hochreiter and Jürgen Schmidhuber.  Long Short-term Memory.  Neural computation, 9:1735–80, December 1997.
  * Krüger et al. [2014] Frank Krüger, Martin Nyolt, Kristina Yordanova, Albert Hein, and Thomas Kirste.  Computational state space models for activity and intention recognition. a feasibility study.  PloS one, 9(11):e109381, 2014.
  * Lampert et al. [2014] Christoph H. Lampert, Hannes Nickisch, and Stefan Harmeling.  Attribute-Based Classification for Zero-Shot Visual Object Categorization.  In IEEE Trans. on Pattern Analysis and Machine Intelligence, volume 36 of 3, pages 453–465. IEEE, 2014.
  * Liaw and Wiener [2002] Andy Liaw and Matthew Wiener.  Classification and regression by randomforest.  R News, 2(3):18–22, 2002.
  * Moya Rueda and Fink [2018] Fernando Moya Rueda and Gernot A Fink.  Learning attribute representation for human activity recognition.  In 2018 24th Int. Conf. on Pattern Recognition (ICPR), pages 523–528. IEEE, 2018.
  * Moya Rueda et al. [2019] Fernando Moya Rueda, Stefan Lüdtke, , Max Schröder, Kristina Yordanova, Thomas Kirste, and Gernot A. Fink.  Combining symbolic reasoning and deep learning for human activity recognition.  In 15th Workshop CoMoRea. in press, 2019.
  * Niemann et al. [2020a] Friedrich Niemann, Christopher Reining, Fernando Moya Rueda, Erik Altermann, Nilah Ravi Nair, Janine Anika Steffens, Gernot A. Fink, and Michael ten Hompel.  Logistic Activity Recognition Challenge (LARa) – A Motion Capture and Inertial Measurement Dataset, 2020.
  * Niemann et al. [2020b] Friedrich Niemann, Christopher Reining, Fernando Moya Rueda, Nilah Ravi Nair, Janine Anika Steffens, Gernot A. Fink, and Michael ten Hompel.  LARa: Creating a Dataset for Human Activity Recognition in Logistics Using Semantic Attributes.  MDPI Sensors, 2020.
  * Ordóńez and Roggen [2016] Francisco Ordóńez and Daniel Roggen.  Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition.  Sensors, 16(1):115, January 2016.
  * Qian et al. [2019] Hangwei Qian, Sinno Jialin Pan, Bingshui Da, and Chunyan Miao.  A novel distribution-embedded neural network for sensor-based activity recognition.  In IJCAI, pages 5614–5620, 2019.
  * R Core Team [2020] R Core Team.  R: A Language and Environment for Statistical Computing.  R Foundation for Statistical Computing, Vienna, Austria, 2020.
  * Ramírez and Geffner [2010] Miguel Ramírez and Hector Geffner.  Probabilistic plan recognition using off-the-shelf classical planners.  In Proc. of the AAAI Conf. on Artificial Intelligence, volume 24, 2010.
  * Reining et al. [2018a] Christopher Reining, Fernando Moya Rueda, Michael Ten Hompel, and Gernot A. Fink.  Towards a Framework for Semi-Automated Annotation of Human Order Picking Activities Using Motion Capturing.  In FedCSIS, pages 817–821. IEEE, 2018.
  * Reining et al. [2018b] Christopher Reining, Fernando Moya Rueda, Michael Ten Hampel, and Gernot A Fink.  Towards a framework for semi-automated annotation of human order picking activities using motion capturing.  In 2018 FedCSIS, pages 817–821. IEEE, 2018.
  * Reining et al. [2018c] Christopher Reining, Michelle Schlangen, Leon Hissmann, Michael ten Hompel, Fernando Moya, and Gernot A Fink.  Attribute representation for human activity recognition of manual order picking activities.  In Proc. of the 5th iWOAR, pages 1–10, 2018.
  * Ronao and Cho [2015] Charissa Ann Ronao and Sung-Bae Cho.  Deep Convolutional Neural Networks for Human Activity Recognition with Smartphone Sensors.  In Sabri Arik, Tingwen Huang, Weng Kin Lai, and Qingshan Liu, editors, Neural Information Processing, Lecture Notes in Computer Science, pages 46–53. Springer Int. Publishing, 2015.
  * Yang et al. [2015] Jianbo Yang, Minh Nhut Nguyen, Phyo Phyo San, Xiaoli Li, and Shonali Krishnaswamy.  Deep convolutional neural networks on multichannel time series for human activity recognition.  In IJCAI, pages 3995–4001, 2015.
  * Yao et al. [2018] Rui Yao, Guosheng Lin, Qinfeng Shi, and Damith C. Ranasinghe.  Efficient dense labelling of human activity sequences from wearables using fully convolutional networks.  Pattern Recognition, 78:252–266, June 2018.
  * Zeng et al. [2014] Ming Zeng, Le T. Nguyen, Bo Yu, Mengshoel Ole J., Jiang Zhu, Pang Wu, and Joy Zhang.  Convolutional neural networks for human activity recognition using mobile sensors.  In MobiCASE, pages 197–205. IEEE, 2014.
  * Zheng et al. [2017] Jingjing Zheng, Zhuolin Jiang, and Rama Chellappa.  Submodular attribute selection for visual recognition.  In IEEE Trans. on Pattern Analysis and Machine Intelligence, volume 39 of 11, pages 2242–2255. IEEE, 2017.


# Shai: A large language model for asset management

Zhongyang Guo  China Asset Management Co., Ltd.  Guanran Jiang  China Asset
Management Co., Ltd.  Zhongdan Zhang  China Asset Management Co., Ltd.  Peng
Li  China Asset Management Co., Ltd.  Zhefeng Wang  China Asset Management
Co., Ltd.  Yinchun Wang Corresponding author e-mail<EMAIL_ADDRESS>China
Asset Management Co., Ltd.

###### Abstract

This paper introduces ”Shai” a 10B level large language model specifically
designed for the asset management industry, built upon an open-source
foundational model. With continuous pre-training and fine-tuning using a
targeted corpus, Shai demonstrates enhanced performance in tasks relevant to
its domain, outperforming baseline models. Our research includes the
development of an innovative evaluation framework, which integrates
professional qualification exams, tailored tasks, open-ended question
answering, and safety assessments, to comprehensively assess Shai’s
capabilities. Furthermore, we discuss the challenges and implications of
utilizing large language models like GPT-4 for performance assessment in asset
management, suggesting a combination of automated evaluation and human
judgment. Shai’s development, showcasing the potential and versatility of
10B-level large language models in the financial sector with significant
performance and modest computational requirements, hopes to provide practical
insights and methodologies to assist industry peers in their similar
endeavors.

##  1 Introduction

Recent advancements in Large Language Models (LLMs) have resulted in
breakthroughs, with 100B-level models like GPT-4 [1], LLaMa2 [2], ChatGLM[3],
BLOOM[4], Falcon[5] and PaLM2[6] leading the way in natural language
processing (NLP) capabilities. These models have shown an exceptional ability
to generate natural and coherent text, understand complex contexts, and adapt
to a wide variety of tasks and scenarios. Besides the general LLM development,
domain specific LLM development is also flourishing, where the domains span
from law[7; 8; 9] to health care[10; 11; 12; 13] and finance[14; 15; 16; 17]
etc. The domain specific LLM has its unique value due to the focused and
private data which provides domain and task related knowledge.

In this work, we introduce Shai, a large language model focusing on asset
management(AM) area. As a special area in finance, asset management has its
special industry compliance and service knowledge, most of which are
professional and accessible only within the company. Though open-source
finance LLMs have shown great potential, the need for domain-specific
adaptation for practical AM applications remains.

Our endeavor for building an AM LLM are as follows:

  * •

First, we pick up and define several NLP tasks for a typical asset management
company, and build the corresponding task dataset for training and evaluation.

  * •

Second, we conduct continuous pretraining and supervised finetuning on a
10B-level base LLM model, providing an optimal balance between performance and
inference cost.

  * •

Third, we conduct evaluation covering our proposed AM tasks. These evaluations
include financial professional examination questions, open Q&As based on real-
world scenarios, specific tasks designed for asset management scenarios, and
safety assessments, providing a comprehensive and objective evaluation. To
gain valuable insights into the relative performance of these models in the
specific context of asset management, we notably bring Shai into direct
comparison with mainstream 10B-level open-source models, such as
baichuan2[18], Qwen[19], InterLM[20], and Xverse[21], on our proprietary
dataset. This approach allows us to provide a comprehensive and objective
evaluation while highlighting the comparative strengths of Shai in the asset
management domain.

Our contributions are: 1) As far as we know, we are the first to build a 10B
level LLM for asset management, which achieve the best performance comparing
to the mainstream 10B-level LLMs. 2) We share our detailed construction
process consisting of continuous training and SFT. 3) We present a few
interesting findings: The LLM model, which appears to be associated with task-
related pre-training strategies, exhibits an advantage in downstream tasks;
The evaluation based on GPT4 has bias on input position and text length.

##  2 LLMs in Asset Management

Figure 1: Asset management business scenarios

Asset Management is a specialized field that offers well-rounded financial
solutions to both individual and institutional investors. Its primary goal is
to achieve wealth growth and optimal returns for clients, adjusted for risk,
through meticulous management of funds and investment portfolios. This field
incorporates several key processes such as investment and market research,
formulating investment strategies, optimizing investment portfolios, risk
management, customer service, and other support and operational tasks.

The complex and multifaceted nature of asset management has amplified the
demand for advanced AI solutions. With the fast-paced advancements in big data
and AI technology, the use of Large Language Models (LLMs) in asset management
has been expanding. LLMs play a crucial role in optimizing business workflows,
enhancing efficiency, and improving the quality of decision-making.

In investment research, for instance, LLMs can assist asset management firms
in quickly and accurately extracting key information from a vast array of
market data, financial reports, and macroeconomic indicators. They can analyze
and summarize this complex information, enabling faster data collation and
reducing errors that can occur due to human intervention.

In the realm of risk management, LLMs can aid asset management companies in
predicting and evaluating various types of risks via sophisticated data
analysis and pattern recognition. For example, when it comes to assessing the
market volatility of a particular asset class, LLMs can swiftly analyze
historical trends and relevant news reports, providing both quantitative and
qualitative support to the risk assessment process.

In customer service and consultation, the application of LLMs has
significantly improved the user interaction experience. They can comprehend
the specific needs and situations of customers, providing targeted responses
or recommendations, which greatly enhances customer satisfaction.

In the context of regulatory compliance, LLMs can interpret complex regulatory
documents, assisting asset management companies in ensuring that their
business operations meet a variety of legal requirements. For instance, when
new financial regulations are introduced, LLMs can quickly summarize the main
changes and potential impacts, helping the company adapt swiftly to changes in
the legal environment. Figure 1 illustrates some specific tasks in the asset
management field where LLMs can be applied.

##  3 Data

The quality and relevance of data play a crucial role in the successful
training of large language models. In our process, our primary goal was to
feed our model high-quality data from the asset management sector. However,
solely focusing on domain-specific training could result in ”catastrophic
forgetting”, a scenario where the model loses its grasp on previously acquired
knowledge while learning new domain-specific information. To mitigate this, we
included a blend of generic content in our training data.

###  3.1 Pre-training Data

During the pre-training phase, we selected a diverse range of data sources for
model training, including textbooks from the financial and economic sector,
research reports, interview records of fund managers, articles from official
Chinese media outlets, and content from encyclopedias, books from various
fields, and corpose from online forums.

Figure 2: Data distribution

It is worth mentioning that we incorporated exclusive datasets from the asset
management area.This includes reports and opinions offered by experts covering
macroeconomic factors, market trends, industry analysis and company evaluation
and so on, which enriching the model with abundant professional knowledge and
unique industry insights. Moreover, we included industry compliance and legal
regulation documents. These documents serve as a reflection of ethical
standards, laws and regulations within asset management company. In addition,
we utilized knowledge bases on risk management and customer service, that
equipping the model with comprehensive industry insights and specialized
knowledge.

However, we must acknowledge the potential errors during data processing, as
both data parsing abilities and OCR systems may make mistakes. Moreover,
online information can contain low-value content. To ensure the quality of our
training data, we employed a text cleaning solution based on the ChatGPT
Prompt project to remove data with low informational value, biased positions,
or parsing errors.

###  3.2 Supervised finetuning data

Our data for Supervised Fine-tuning was divided into four parts: general
dialogue, financial vertical Q&A, asset management tasks, and proprietary
industry data.

  * •

For the general dialogue portion, we utilized open-source data from
Alpaca[22], RefGPT[23], and sharegpt[24]. The Alpaca and RefGPT data have high
accuracy and were used directly. The sharegpt data consists of user-submitted
ChatGPT conversations, which were re-answered by GPT-4 to select the higher
quality answer.

  * •

In the asset management field Q&A portion, we generated a series of question-
answer pairs by having GPT-4 read materials from the financial field. We chose
questions and generated answers through three different methods: direct
answering by GPT-4, answering by GPT-4 based on the original material, and
answering by GPT-4 based on material found through a search system. The best
answer was selected by GPT-4 for training data.

  * •

We also designed specific financial tasks for the asset management field to
enhance the model’s abilities in understanding, extraction, analysis, and
logical reasoning. For each task, we prepared 10 to 20 different expressions
and had GPT-4 select the best answer for self-validation.

  * •

Lastly, we use proprietary natural dialogue data within our company. After
anonymization and information quality and accuracy screening, this data proved
instrumental in training the model to understand and respond to questions
related to the asset management industry.

After all of the processing, we had approximately 75k samples for Supervised
Fine-tuning.

##  4 Training

In the training phase of our study, we adopted a flexible approach, selecting
an open-source language model as our foundational model, with the
understanding that this choice may evolve based on future developments. We
found that some open-source models are capable of generating content that
aligns perfectly with the format of actual test questions during their text
completion tasks, as shown in Table 1. We infer that these models probably
have taken the task related corpus during pre-training instead of using only
general unsupervised text. Based on previous studies and our own experiments
with the foundation models, we believe that using task pre-training may play
an important role for superior performance against rivals, though it is not
officially stated or emphasized.

Prompt |  在间接标价法下，当外汇远期汇率出现贴水现象，它会
---|---
Model 1 (with task related corpus ) |  （ ）。A.提高进口成本 B.降低进口成本 C.提高出口收益 D.降低出口收益\n\n B 在间接标价法下，当外汇远期汇率出现贴水现象，说明远期汇率低于即期汇率，这会使进口成本降低。
Model 2 (with general unsupervised text) |  促使进口商或出口商进行套期保值交易。在直接标价法下，当外汇远期汇率出现升水现象，它会促使进口商或出口商进行套期保值交易。\n \n远期汇率#远期汇率的作用\n远期汇率是外汇市场上的一种重要的金融工具，它为外汇的买卖双方提供了规避汇率风险的手段

Table 1: Comparison of generation outcomes between a model trained with task
related corpus and a model trained with general unsupervised text.

Our model adopts a structure that utilizes a ChatML[25] template. This
approach uses natural language along with special characters to denote the
question-answer relationship between the user and the AI. To enhance this, we
implement the concept of prompt-tuning[26; 27], increasing the number of
special characters to 10. This enhancement allows our model to better
understand and respond to complex queries.

During the pre-training phase, we used a natural Q&A corpus from the internet,
particularly from zhihu.com. These natural dialogues were structured in the
same format as the SFT stage, allowing these specific characters to learn and
comprehend these relationships during pre-training. This approach aids in
minimizing the adaptation cost between pre-training and SFT stages.

##  5 Evaluations

###  5.1 Evaluations tasks

To comprehensively assess the performance of large language models in the
asset management industry, we have constructed a multi-faceted evaluation
dataset. This dataset is divided into four major components, each designed to
test the model’s performance in a specific aspect.

  * •

Firstly, we employ financial professional examination questions to evaluate
the model’s financial knowledge. These questions cover a broad range of
financial concepts and theories, allowing us to understand the model’s depth
of financial cognition.

  * •

Secondly, open Q&A sessions related to asset management business are used to
evaluate the model’s ability to understand complex queries and generate
knowledgeable responses. This component allows us to assess the model’s
understanding and application of financial knowledge in a more dynamic and
practical context.

  * •

Thirdly, we have designed specific tasks for asset management scenarios. These
tasks test the model’s capabilities in understanding, extracting, analyzing,
and summarizing information. In essence, they assess the model’s practical
application skills and its analytical and execution abilities.

  * •

Lastly, we conduct safety assessments to evaluate the model’s capabilities in
adhering to economic safety and compliance standards within the asset
management field. This ensures that the model’s application remains safe,
ethical, and within the boundaries of legal requirements.

Figure 3: Asset management domain large language model evaluation framework

These four parts of data constitute a comprehensive and rigorous assessment
framework for large language models within the context of asset management.
Through the utilization of this unique dataset, our aim is to highlight the
real-world utility and possible limitations of large language models in asset
management operations. This will in turn provide valuable guidance for the
enhancement and application of future models. In total, we devised 6377
evaluation questions spanning 24 sub-tasks. Table 2 provides an detailed
overview of these specific tasks.

Task |  Description
---|---
Investment Research Q&A |  Q&A related to investment research, including macroeconomics, industry, company, etc.
Investment Advisory Q&A |  Q&A related to investment advisory issues, including investment portfolio, asset allocation, investment consulting, investment management, etc.
Legal Regulation Q&A |  Q&A related to financial regulations, including various laws and policies.
Risk Management Q&A |  Q&A related to risk control case analysis and rule interpretation.
Customer Service Q&A |  Q&A related to real customer service questions.
Mathematical Questions(FMQ) |  Perform financial mathematical calculations, including interest rate, valuation calculation, etc.
Financial Data Q&A(FD-Q&A) |  Answer questions based on background information.
Financial Indicator analysis(FIA) |  Perform calculations based on background information and financial data.
Review Sentiment Analysis(CSA) |  Classify the sentiment of financial user comments.
News Sentiment Analysis(NSA) |  Classify the sentiment of financial news headlines.
Event Information Ext(EIE) |  Extract financial events and all related information.
Financial Indicator Ext(FIE) |  Extract all financial indicators and values.
Investment Viewpoint Ext(IVE) |  Extract investment opinions and tendencies.
Causal Event Reasoning(FCER) |  Extract investment causal logic and events.
News Summary(NS) |  Summarize and generate headlines for financial news.
Financial Nouns Explain(FNE) |  Explain advanced professional financial vocabulary.
General Safety |  General safety issues, including prejudice, discrimination, crime, network safety and other areas.
Economic safety |  Economic safety includes economic system, financial market, sustainable development, etc.
AM compliance |  Compliance mainly refers to the internal code of conduct and ethical standards of asset management companies.
Table 2: The detailed description of the evaluation task (As financial
professional exams consist of standard multiple-choice questions, which are
not further elaborated here).

###  5.2 Scoring methods

To ensure the fairness and objectivity of the evaluation, we adopted the Zero-
Shot mode for all evaluation questions. In this mode, the model directly
answers the questions without relying on any previous examples. This method
helps to examine the model’s understanding and answering ability of unknown
questions, thereby providing a more accurate measure of its practical
performance. To reduce the impact of randomness in the evaluation process, we
conducted five independent evaluations for each model, averaging the scores to
determine the final score. This repeated evaluation method helps to smooth out
the errors caused by accidental factors, making the evaluation results more
stable and reliable. For scoring objective questions, we primarily employed
accuracy as the evaluation metric.

In the open Q&A part, we initially explored GPT-4’s scoring ability. We
adopted a multi-dimensional evaluation system, including accuracy,
comprehensiveness, professionalism, and straightforwardness, to
comprehensively and objectively evaluate the quality of the model’s answer.
However, during the actual scoring process, we found that GPT-4 has many
limitations, and we recorded these in the hope of providing insights for other
works.

  * •

Position Bias: Building on the discussion in previous research like Wang’s[28]
about the effect of answer order in large models, we carried out an
investigation to validate this order effect and proposed a more refined
approach for determining the winner. To verify this hypothesis, we applied the
Wilcoxon[29] signed-rank test to analyze the impact of order changes on model
scores. The test results showed an effect size $r$ value of -0.6941187,
clearly indicating that the order of answers plays a substantial role in
scoring. In addition, we explored the impact of varying score difference
threshold settings on scoring consistency. We found that the higher the score
difference threshold, the higher the consistency of scoring results (shown in
figure 4). Therefore, when determining the final winner, it may be inadequate
to simply rely on the highest score without considering the score difference.
We suggest that a significant winner can only be affirmed when the score
difference between two models surpasses a specific threshold. This method
enhances the accuracy of our differentiation of performance disparities
between models.

  * •

Length Bias: Our study indicates that GPT-4 seems to favor longer answers
during the scoring process, which is consistent with previous findings on
verbosity bias in large language models [30; 31]. However, the influence of
length on scoring is subtle and multifaceted. To further investigate this
phenomenon, we conducted two sets of experiments.

In the first experiment, we explored the overall impact of length on scoring,
not just focusing on this single variable. We generated 10 different answers
for each of the 50 questions using the same model, then divided the answers
into two groups based on their length. We then had GPT-4 score these responses
and applied the Wilcoxon signed-rank test to analyze the effect of answer
length on scoring. The results showed a significant difference between the two
groups ($p<0.001$), with longer answers receiving higher average scores (9.67)
than shorter ones (9.13). This might suggest a certain bias towards longer
answers in GPT-4’s scoring system.

In the second experiment, we controlled for the amount of information in the
answers and focused on exploring the impact of verbosity. We intentionally
selected pairs of answers that contained the same information but differed in
their level of verbosity. Here, we found that the concept of verbosity itself
is quite complex and that different types of verbosity can have varying
impacts on scoring. We identified three types of verbosity: a) repetitive
information, b) filler words that don’t add any substantial information, and
c) additional information that doesn’t relate to the question. Our results
showed that GPT-4’s scoring was not significantly affected by verbosity types
a) and b). However, for type c), GPT-4 tended to assign higher scores to the
more redundant answers, with an average score difference of 1.14 points.

  * •

Domain-Specific Knowledge Limitations: GPT-4 showed a certain error rate in
evaluating queries within the asset management domain, signifying its
limitations in knowledge acquisition and application in this specific area. As
shown in the example below, GPT-4 misunderstood the concept of ”日光基” (a term
describing a very popular fund that can be sold out in one day), leading to an
incorrect judgment during the evaluation. Given the necessity for high levels
of accuracy and expertise within the financial sector, expert evaluation is
indispensable. We assembled a team of specialists to assess open Q&A, which
served as the ultimate evaluation criterion.

Figure 4: Score difference thresholds and score consistency relationship.
Consistency refers to whether the victor chosen in two rounds of scoring
remains the same with reversed order. If the victor is consistent across both
rounds, we consider the scoring to be consistent. Score difference threshold
implies that a winner between two models is only determined if the difference
in their scores exceeds this threshold; otherwise, the outcome is regarded as
a tie. Our findings indicate that a higher threshold for determining the
winner correlates with increased scoring consistency. Figure 5: GPT-4
misrating case

###  5.3 Evaluation result

We assessed the performance of financial exam questions using two distinct
evaluation methods: Chain of Thought (COT) [32] and Answer Only (AOT). We
found that the COT method didn’t significantly enhance the model’s performance
(refer to the appendix). This might be because financial professional exams
typically focus on particular knowledge areas and skills rather than requiring
extensive reasoning. Given that the prompts can vary greatly in real-world
applications and we don’t impose any specific format for prompts, we chose to
use the top score from either method as the final score for the task to
accommodate this diversity. In comparison to other models of a similar size
(around 1.3 billion parameters), our model topped the leaderboard in the task
of financial professional exams. This suggests that our model is more accurate
in dealing with professional exams.

Task |  Shai-14B |  Qwen-14B |  Baichuan 2-13B |  InternLM-20B |  XVERSE-13B |  GPT-3.5 |  Gpt 4
---|---|---|---|---|---|---|---
Fund |  75.5 |  69.6 |  53.2 |  54.3 |  54.3 |  52.1 |  72.0
Securities |  78.0 |  74.6 |  63.0 |  59.4 |  60.5 |  62.0 |  79.9
Banking |  78.5 |  72.4 |  58.9 |  56.0 |  58 |  57.6 |  77.9
Futures |  59.3 |  53.8 |  44.8 |  38.3 |  44.0 |  43.9 |  62.4
CFA |  53.9 |  52.3 |  43.9 |  46.4 |  44.2 |  49.7 |  62.3
Table 3: Scores for financial exam tasks(the maximum value in AOT and COT)

When evaluating specific task practices in asset management scenarios, our
model displayed a strong practical application ability. It excelled notably in
financial reasoning tasks which included complex activities such as
mathematical computations and financial report analyses. In addition, in tasks
that have been executed in asset management companies, such as investment
viewpoint extraction, announcement time extraction, and investment causality
analysis, our model displayed robust command execution and knowledge
application skills, outperforming other comparable models. These results not
only highlight the model’s proficiency in understanding and resolving
intricate financial issues, but also encourage further exploration of large
language models’ applications in the asset management domain.

However, in some application scenarios that do not have significant financial
characteristics, such as news summary generation, our model did not exhibit
particularly outstanding performance. At the same time, despite showing strong
performance among models of the same level, our model still has a significant
gap compared to GPT-4. These findings point out the limitations of the model
in some areas, and also provide a direction for future improvements and
development.

Task |  Shai-14B |  Qwen-14B |  Baichuan 2-13B |  InternLM-20B |  XVERSE-13B |  GPT-3.5 |  GPT-4
---|---|---|---|---|---|---|---
FMQ |  37.7 |  36.7 |  33.4 |  31.1 |  24.6 |  39.7 |  57.6
FD-Q&A |  95.5 |  93.5 |  90.8 |  83.5 |  83.8 |  94.5 |  97.5
FIA |  50.5 |  33.8 |  36.4 |  20.7 |  16.7 |  59.1 |  75.4
CSA |  76.7 |  77.3 |  72.0 |  64.0 |  54.0 |  72.0 |  84.7
NSA |  95.2 |  95.2 |  78.4 |  86.4 |  92.8 |  82.0 |  97.6
EIE |  83.7 |  71.6 |  64.6 |  68.2 |  56.0 |  79.8 |  91.9
FIE |  88.2 |  84.9 |  74.4 |  77.3 |  82.3 |  77.1 |  95.8
IVE |  75.3 |  68.6 |  70.2 |  66.9 |  64.7 |  73.7 |  87.9
FCER |  88.4 |  77.5 |  81.4 |  62.2 |  59.7 |  87.3 |  93.1
NS |  75.3 |  66.3 |  77.16 |  56.8 |  79.7 |  - |  85.3
FNE |  83.0 |  79.3 |  74.0 |  54.7 |  61.3 |  77.7 |  89.3
Table 4: Scores for AM scenario application tasks (The scoring method for ns
tasks is to calculate the non-negative ratio compared to gpt3.5, so gpt3.5
does not show the score.)

Regarding safety, all the assessed models demonstrated substantial efficacy in
general safety metrics such as discrimination, propensity towards violence,
health data management, fairness, and network safety. Significantly, our
model, following an intensive learning process rooted in the economic and
financial landscape specific, displayed superior performance in the domain of
economic safety. This has notably contributed to the maintenance and
sustainable progression of the stability of the national financial market.
Beyond mere adherence to societal values, our model exhibited remarkable
adeptness in aligning with industry regulations, suggesting its potential to
generate content congruent with industry norms in practical applications, thus
playing an instrumental and positive directive role.

Safety Task |  Shai-14B |  Qwen-14B |  Baichuan 2-13B |  InternLM-20B |  XVERSE-13B |  GPT-3.5 |  GPT-4
---|---|---|---|---|---|---|---
General |  96.8 |  93.7 |  96.4 |  96.0 |  93.7 |  90.4 |  94.7
Economic |  98.0 |  94.4 |  87.3 |  95.0 |  91.0 |  71.8 |  82.0
Compliance |  82.5 |  76.5 |  63.4 |  67.3 |  72.0 |  62.2 |  79.8
Table 5: Scores for safety tasks

In subjective Q&A questions, after evaluation by the expert team, our model
emerged as the top performer amongst comparable models.

Modle |  Shai-14B |  Qwen-14B |  Baichuan 2-13B |  InternLM-20B |  XVERSE-13B |  GPT-3.5 |  GPT-4
---|---|---|---|---|---|---|---
Score |  74.7 |  72.6 |  66.4 |  37.9 |  48.4 |  71.1 |  84.3
Table 6: Scores for open Q&A tasks

We found that our model has effectively assimilated a broad spectrum of
financial knowledge via the pre-training phase, thereby enriching its
foundational knowledge base. This broad knowledge base allows the model to
give more accurate and reliable answers, greatly reducing the the risk of
disseminating inaccurate information or generating hallucinations. For
instance, concerning the concept of ”北向资金” some large language models
incorrectly interpret it as ”funds flowing from mainland China into Hong
Kong.” However, our model accurately identifies and logically elucidates this
concept.

Figure 6: Case 1 of open Q&A

Our model also exhibited exceptional proficiency in financial reasoning. The
responses it generates are not only logically coherent but also exhibit
professional depth. Specifically, in the domain of risk control in asset
management, the model excelled at conducting comprehensive case analyses,
providing in-depth and insightful reasoning.

Figure 7: Case 2 of open Q&A

Moreover, our model has exhibited an impressive capacity for flexibility. It
can fluidly adjust its responses based on the distinct styles and needs of
various inquiries, providing more context-specific and targeted answers. Thus,
whether responding to queries from professional financial analysts or
financial consumers, the model can generate responses that are aptly suited to
their particular requirements. As illustrated in Figure 8 and Figure 9 , this
flexible capability significantly enhances the model’s practicality and
effectiveness in real-world applications, enabling it to excel in complex
financial analysis tasks, while also playing a crucial role in customer
service.

Figure 8: Case 3 of open Q&A Figure 9: Case 4 of open Q&A

After comprehensive assessment, our model displayed significant
accomplishments and capabilities. In the financial examination component, the
model performed admirably, indicating its extensive knowledge base. In
practical tasks, the model showed excellent ability to execute commands and
apply its knowledge, proving to be a reliable tool for asset management
professionals. In the business Q&A section, the model also displayed a high
level of accuracy and logical consistency, as evaluated by our expert panel.
Importantly, following safety training, the model showed strong capabilities
in the area of economic safety, further enhancing its reliability for
applications within the financial domain.

##  6 Conclusion

In this research, we have developed ”Shai”, a 10B level language model
specifically designed for asset management, leveraging advanced training
techniques and a diverse array of financial data. Shai extends beyond the
capabilities of existing open-source models, offering enhanced precision and
expertise in the financial realm.

Our evaluation framework, specifically designed for this sector, combines
financial exams, open-ended question-answering, practical tasks, and rigorous
safety assessments. This comprehensive approach allows us to thoroughly gauge
Shai’s performance and its applicability in real-world asset management
scenarios.

The results demonstrate that Shai excels not only in financial examinations
and practical tasks but also in critical safety aspects, such as economic
security and adherence to ethical standards. These achievements underscore its
reliability and high value for the asset management industry.

In summary, our study highlights the significant potential of large language
models like Shai in the field of asset management. Moving forward, our efforts
will concentrate on refining Shai’s functionality and exploring broader
applications, thereby enriching the role of AI in asset management and
advancing the field towards a more intelligent and data-driven future.

## References

  * OpenAI [2023] OpenAI.  Gpt-4 technical report, 2023.
  * Touvron et al. [2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023.
  * Zeng et al. [2022] Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al.  Glm-130b: An open bilingual pre-trained model.  _arXiv preprint arXiv:2210.02414_ , 2022.
  * Workshop et al. [2023] BigScience Workshop, :, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, and et al Stella Biderman.  Bloom: A 176b-parameter open-access multilingual language model, 2023\.
  * Almazrouei et al. [2023] Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, et al.  Falcon-40b: an open large language model with state-of-the-art performance.  Technical report, Technical report, Technical report, Technology Innovation Institute, 2023.
  * Anil et al. [2023] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al.  Palm 2 technical report.  _arXiv preprint arXiv:2305.10403_ , 2023.
  * Cui et al. [2023] Jiaxi Cui, Zongjian Li, Yang Yan, Bohua Chen, and Li Yuan.  Chatlaw: Open-source legal large language model with integrated external knowledge bases.  _arXiv preprint arXiv:2306.16092_ , 2023.
  * Nguyen [2023] Ha-Thanh Nguyen.  A brief report on lawgpt 1.0: A virtual legal assistant based on gpt-3.  _arXiv preprint arXiv:2302.05729_ , 2023.
  * Huang et al. [2023] Quzhe Huang, Mingxu Tao, Zhenwei An, Chen Zhang, Cong Jiang, Zhibin Chen, Zirui Wu, and Yansong Feng.  Lawyer llama technical report.  _arXiv preprint arXiv:2305.15062_ , 2023.
  * Xiong et al. [2023] Honglin Xiong, Sheng Wang, Yitao Zhu, Zihao Zhao, Yuxiao Liu, Qian Wang, and Dinggang Shen.  Doctorglm: Fine-tuning your chinese doctor is not a herculean task.  _arXiv preprint arXiv:2304.01097_ , 2023.
  * Wang et al. [2023a] Haochun Wang, Chi Liu, Nuwa Xi, Zewen Qiang, Sendong Zhao, Bing Qin, and Ting Liu.  Huatuo: Tuning llama model with chinese medical knowledge.  _arXiv preprint arXiv:2304.06975_ , 2023a.
  * Chen et al. [2023a] Yirong Chen, Xiaofen Xing, Jingkai Lin, Huimin Zheng, Zhenyu Wang, Qi Liu, and Xiangmin Xu.  Soulchat: Improving llms’ empathy, listening, and comfort abilities through fine-tuning with multi-turn empathy conversations.  _arXiv preprint arXiv:2311.00273_ , 2023a.
  * Chen et al. [2023b] Yirong Chen, Zhenyu Wang, Xiaofen Xing, Zhipei Xu, Kai Fang, Junhong Wang, Sihang Li, Jieling Wu, Qi Liu, Xiangmin Xu, et al.  Bianque: Balancing the questioning and suggestion ability of health llms with multi-turn health conversations polished by chatgpt.  _arXiv preprint arXiv:2310.15896_ , 2023b.
  * Wu et al. [2023] Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann.  Bloomberggpt: A large language model for finance.  _arXiv preprint arXiv:2303.17564_ , 2023.
  * Zhang and Yang [2023] Xuanyu Zhang and Qing Yang.  Xuanyuan 2.0: A large chinese financial chat model with hundreds of billions parameters.  In _Proceedings of the 32nd ACM International Conference on Information and Knowledge Management_ , pages 4435–4439, 2023.
  * Yang et al. [2023a] Hongyang Yang, Xiao-Yang Liu, and Christina Dan Wang.  Fingpt: Open-source financial large language models.  _arXiv preprint arXiv:2306.06031_ , 2023a.
  * Chen et al. [2023c] Wei Chen, Qiushi Wang, Zefei Long, Xianyin Zhang, Zhongtian Lu, Bingxuan Li, Siyuan Wang, Jiarong Xu, Xiang Bai, Xuanjing Huang, et al.  Disc-finllm: A chinese financial large language model based on multiple experts fine-tuning.  _arXiv preprint arXiv:2310.15205_ , 2023c.
  * Yang et al. [2023b] Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, Fan Yang, et al.  Baichuan 2: Open large-scale language models.  _arXiv preprint arXiv:2309.10305_ , 2023b.
  * Bai et al. [2023] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al.  Qwen technical report.  _arXiv preprint arXiv:2309.16609_ , 2023.
  * Team [2023] InternLM Team.  Internlm: A multilingual language model with progressively enhanced capabilities, 2023.
  * xverse ai [2023] xverse ai.  Xverse-13b: A multilingual large language model developed by xverse technology inc.  https://github.com/xverse-ai/XVERSE-13B, 2023.
  * Taori et al. [2023] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto.  Stanford alpaca: an instruction-following llama model (2023).  _URL https://crfm. stanford. edu/2023/03/13/alpaca. html_ , 1(2):3, 2023.
  * Yang et al. [2023c] Dongjie Yang, Ruifeng Yuan, Yuantao Fan, , Yifei Yang, Zili Wang, and Shusen Wang.  Refgpt: Reference-to-dialogue by gpt and for gpt.  https://github.com/ziliwangnlp/RefGPT, 2023c.
  * Zheng et al. [2023a] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al.  Judging llm-as-a-judge with mt-bench and chatbot arena.  _arXiv preprint arXiv:2306.05685_ , 2023a.
  * OpenAI [2022] OpenAI.  Chatml.  https://github.com/openai/openai-python/blob/e389823ba013a24b4c32ce38fa0bd87e6bccae94/chatml.md, 2022\.
  * Li and Liang [2021] Xiang Lisa Li and Percy Liang.  Prefix-tuning: Optimizing continuous prompts for generation.  _arXiv preprint arXiv:2101.00190_ , 2021.
  * Liu et al. [2022] Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang.  P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 61–68, 2022.
  * Wang et al. [2023b] Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui.  Large language models are not fair evaluators.  _arXiv preprint arXiv:2305.17926_ , 2023b.
  * Cuzick [1985] Jack Cuzick.  A wilcoxon-type test for trend.  _Statistics in medicine_ , 4(1):87–90, 1985.
  * Saito et al. [2023] Keita Saito, Akifumi Wachi, Koki Wataoka, and Youhei Akimoto.  Verbosity bias in preference labeling by large language models.  _arXiv preprint arXiv:2310.10076_ , 2023.
  * Zheng et al. [2023b] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al.  Judging llm-as-a-judge with mt-bench and chatbot arena.  _arXiv preprint arXiv:2306.05685_ , 2023b.
  * Wei et al. [2022] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in Neural Information Processing Systems_ , 35:24824–24837, 2022.

##  7 Appendix*

Question |  宣传的时候业绩都特别好，为什么一买就亏?
---|---
Shai-14B |  首先,宣传中的业绩是基于历史数据,过去的业绩并不能准确预测未来的业绩。其次,股票市场有其固有的风险,市场的波动是不可预测的,投资并非总是能够盈利。尽管我们尽最大努力确保客户了解投资的风险和潜在收益,但投资的回报仍取决于多种因素,包括市场行情、经济环境和公司的经营状况等。最好的投资策略是基于长期的投资目标,多样化投资,定期评估和调整投资组合,以适应市场变化。消费者应该理性看待宣传信息，不要盲目相信宣传，在做出投资决策之前,您可能需要寻求专业的投资建议,了解投资产品和市场风险, 做出明智的投资决策。
Qwen-14B |  可能的原因有很多，以下是一些可能的解释：1. 宣传和销售策略：公司可能会采取一些促销策略来吸引投资者购买其产品或服务。这些策略可能会让投资者感到兴奋和乐观，但并不能保证投资回报。此外，有些公司可能会夸大其产品的效果或收益潜力，以吸引投资者。2. 市场波动：股市和其他投资市场都是波动性的，这意味着价格会不断变动。即使一个公司在宣传时表现出色，也可能会受到市场环境的影响，导致其股价下跌。3. 投资风险：所有的投资都有风险，包括股票、债券、基金等。投资者需要理解并接受这些风险，才能做出明智的…
Baichuan 2-13B |  作为一个大语言模型，我无法确定您所提到的具体产品或服务。但是，一般来说，可能有以下几种原因：1. 夸大宣传：有些公司为了吸引投资者或者消费者，会过分夸大自己的产品和服务的优势和收益，但实际上可能并没有那么好的效果。2. 高风险投资：如果是一种高风险的投资产品，那么在宣传时可能会强调其高收益性，但是在实际购买和操作过程中可能会出现亏损的情况。3. 市场变化：市场环境的变化可能会导致一些产品和服务的效果受到影响，从而导致购买后的实际效果与宣传时的预期不同。4. 个人…
InternLM-20B |  基金公司为了吸引客户购买基金，会进行各种形式的广告宣传。在宣传时，他们会以过去一段时间内该基金的收益情况作为参考，比如最近一年、三年、五年的收益率等。但这些数据并不代表未来收益如何，投资者需要理性看待这些数据，不能盲目跟风。
XVERSE-13B |  在销售或宣传过程中,产品或服务可能会被过度夸大其优点或者预期收益,让消费者产生购买的决定。但实际上,当消费者真正购买并使用时,他们可能会发现与之前所听说的不同,从而导致不满意或者亏损。这是一种常见的销售策略,也就是所谓的”宣传误导”。
GPT-3.5 |  宣传时的业绩表现好，主要是因为公司会通过各种手段来夸大自己的优势和潜力，吸引投资者的注意。但实际上，这些宣传所展示的数据往往是选择性披露的，可能并不能真实反映公司的真实状况。另外，投资股票存在风险，市场行情的波动、公司内部管理问题、行业竞争等因素都可能导致股票价格的波动。即使是宣传业绩表现好的公司，也不一定能够保证股票的价格一直上涨…
GPT-4 |  这可能有几个原因。首先，市场状况是非常不确定的，一个产品或投资可能在某个时期表现得非常好，但这并不意味着它在未来也会有同样的表现。其次，一些公司或销售人员可能会过度夸大产品或投资的潜力，以吸引更多的买家。此外，你可能没有足够的信息或经验来正确评估一个投资的风险。最后，也有可能你购买的时机不佳，比如在价格高峰时购买，然后价格下跌。总的来说，投资需要谨慎，理性分析，不要只听信一面之词，要多方面获取信息，做出明智的决策。
Table 7: Case of Open Q&A for All Models Question |  什么是即付年金？
---|---
Shai-14B |  即付年金是一种定期支付的等额现金流，通常在每个支付周期开始时支付。即付年金与普通年金的区别在于付款时间的不同。在计算即付年金的终值和现值时，可以利用后付年金系数表，在普通年金的基础上用终值和现值的计算公式进行调整。这种支付方式常见于养老金、租金、分期付款等。例如，如果一个人每月收到1000元的养老金，那么这1000元就是他的即付年金。
Qwen-14B |  即付年金是一种特殊的年金形式，它是指每期期末支付的等额款项。也就是说，每期支付的金额是在该期期末支付的，而不是在该期期初支付的。这种年金的形式通常用于投资或贷款，因为它可以提供稳定的现金流，可以帮助投资者或借款人更好地管理他们的财务状况。
Baichuan 2-13B |  即付年金（Annuity due）是一种金融术语，指的是一种特殊类型的年金支付。年金是指在固定的时间间隔内，连续支付一定金额的资金安排。即付年金的特点是每期支付的金额相等，且第一期付款发生在当前，而不是在未来的某个时间点。
InternLM-20B |  即付年金是指在购买保险产品时，投保人可以选择按月或按季支付保费，保险公司则会在合同生效后的第一个保单周年日向被保险人支付相应的年金。
XVERSE-13B |  即付年金是一种保险形式,在购买时一次性支付一笔年度保费。这种类型的保险主要针对退休人群,旨在为他们提供稳定的收入流。当保单持有人达到一定年龄或保单期满时,保险公司开始每年向其支付保费,直到他们去世或保单期满。这使得保单持有人可以在退休后获得稳定的收入,而不必担心未来可能出现的经济问题。
GPT-3.5 |  即付年金是一种保险产品，指的是在购买保险时一次性支付全部保费，并在一定的保险期限内，每年按照约定的金额获得固定的年金收益。即付年金通常用于退休规划，保证投保人在退休后能够获得稳定的收入。投保人可以根据自己的需求选择保险期限和年金金额，以满足个人的退休生活需求。
GPT-4 |  即付年金，又称即时年金，是一种年金保险产品。它是指投保人向保险公司一次性交付保费后，保险公司立即按照约定的频率向投保人或者受益人支付年金，直到投保人或者受益人死亡为止。这种年金支付方式可以为投保人提供稳定的收入来源，常被用于养老金规划中。
Table 8: Case of Open Q&A for All Models |  Task |  Shai-14B |  Qwen-14B |  Baichuan 2-13B |  InternLM-20B |  XVERSE-13B |  GPT-3.5 |  GPT-4
---|---|---|---|---|---|---|---|---
AOT |  Fund |  75.5 |  69.6 |  53.2 |  54.3 |  54.3 |  52.5 |  70.4
|  securities |  78.0 |  73.8 |  60.9 |  59.4 |  59.4 |  60.0 |  79.9
|  Banking |  78.5 |  72.4 |  58.9 |  56.0 |  56.6 |  57.6 |  77.9
|  Futures |  59.3 |  51.8 |  44.3 |  37.5 |  42.5 |  43.9 |  60.7
|  CFA |  52.7 |  51.1 |  43.1 |  46.4 |  42.4 |  49.4 |  60.9
COT |  Fund |  74.1 |  69.0 |  53.0 |  53.5 |  52.9 |  52.1 |  72.0
|  securities |  76.0 |  74.6 |  63.0 |  55.4 |  60.5 |  62.0 |  76.0
|  Banking |  76.6 |  69.3 |  57.0 |  52.1 |  58.0 |  56.8 |  75.5
|  Futures |  58.6 |  53.8 |  44.8 |  38.3 |  44.0 |  42.4 |  62.4
|  CFA |  53.9 |  52.3 |  43.9 |  42.7 |  44.2 |  49.7 |  62.3
Table 9: Scores for Financial Exam Tasks(AOT and COT)


August 27, 2024

The DUNE Collaboration