# Multimodal sensor data fusion for in-situ classification of animal behavior
using accelerometry and GNSS data

Reza Arablouei  Ziwei Wang  Greg J. Bishop-Hurley  Jiajun Liu  Data61, CSIRO,
Pullenvale QLD 4069, Australia  Agriculture and Food, CSIRO, St Lucia QLD
4067, Australia

###### Abstract

In this paper, we examine the use of data from multiple sensing modes, i.e.,
accelerometry and global navigation satellite system (GNSS), for classifying
animal behavior. We extract three new features from the GNSS data, namely,
distance from water point, median speed, and median estimated horizontal
position error. We combine the information available from the accelerometry
and GNSS data via two approaches. The first approach is based on concatenating
the features extracted from both sensor data and feeding the concatenated
feature vector into a multi-layer perceptron (MLP) classifier. The second
approach is based on fusing the posterior probabilities predicted by two MLP
classifiers. The input to each classifier is the features extracted from the
data of one sensing mode. We evaluate the performance of the developed
multimodal animal behavior classification algorithms using two real-world
datasets collected via smart cattle collar tags and ear tags. The leave-one-
animal-out cross-validation results show that both approaches improve the
classification performance appreciably compared with using data of only one
sensing mode. This is more notable for the infrequent but important behaviors
of walking and drinking. The algorithms developed based on both approaches
require little computational and memory resources hence are suitable for
implementation on embedded systems of our collar tags and ear tags. However,
the multimodal animal behavior classification algorithm based on posterior
probability fusion is preferable to the one based on feature concatenation as
it delivers better classification accuracy, has less computational and memory
complexity, is more robust to sensor data failure, and enjoys better
modularity.

###### keywords:

animal behavior classification , accelerometer data , deep learning , embedded
systems , global navigation satellite system , information fusion , multimodal
sensing , on-device inference.

C[1]>p#1

##  1 Introduction

Accurate knowledge of the behavior of livestock is important for monitoring
and managing their health, welfare, and productivity. However, long-term
direct observation of behavior is challenging, particularly for large numbers
of animals spread over large areas. This has motivated extensive research on
the use of wearable sensors for automated remote recognition of various animal
behaviors. Micro-electromechanical systems (MEMS) inertial measurement unit
(IMU) sensors, particularly accelerometers, are among the most popular sensors
with small form factor and low power consumption that have been successfully
utilized for classification of animal behavior. Examples include the works of
(Kamminga et al., 2018; Fogarty et al., 2020; Brandes et al., 2021; Pavlovic
et al., 2021) and the references therein.

The global navigation satellite system (GNSS) receiver is another device that
is commonly used as a wearable sensor on animals of various species and sizes,
both livestock and wildlife, to localize and track their positions. There have
been some reports in the literature on using the data produced by GNSS
receivers alone to classify animal behavior, for example, in (de Weerd et al.,
2015; Williams et al., 2016; Suparwito et al., 2021; Roy et al., 2022). Most
of these works, calculate features from GNSS data over time segments that are
significantly longer than those typically used when extracting features from
IMU data. For instance, in (Williams et al., 2016), it is shown that using
segments of 160 seconds results in better performance compared with using
shorter time segments. Meanwhile, accelerometry features used for animal
behavior classification are often extracted from the data of time segments
smaller than 15 seconds (Riaboff et al., 2022). This difference is partly due
to the frequency of GNSS data acquisition being set to much lower values
compared with the sampling rate of accelerometer sensors as GNSS receivers
consume significantly more power. The highest rate in most commercial GNSS
receivers is 1 Hz while the sampling rate for MEMS accelerometers can be up to
several KHz with typical values used for animal behavior classification being
around 10 to 50 Hz (Riaboff et al., 2022). The amount of discriminative
information in the data and the associated uncertainty are other key factors
determining the time segment size. The features that are commonly extracted
from GNSS data for animal behavior classification are based on the distance
traveled by the animal and the rate of change in the heading measurements,
which relate to the direction of animal’s movement. In addition, the range of
behaviors that can be classified using GNSS data is usually more limited
compared with the behaviors that can be classified using accelerometry data.

Accelerometry and GNSS data can provide discriminative knowledge regarding
animal behaviors of interest that complement each other. This is because they
may carry distinct information about the behaviors while being governed and
affected by statistically independent processes. Some researchers have
explored the advantages of the combined use of accelerometry and GNSS data for
animal behavior classification (Bailey et al., 2018; Brennan et al., 2021;
Fogarty et al., 2021; Cabezas et al., 2022). In (Fogarty et al., 2021),
several features extracted from accelerometry, GNSS, and weather data are used
together with a support vector machine classifier to recognize behaviors that
are relevant to detecting parturition events in grazing ewes. In (Cabezas et
al., 2022), the accelerometry data is used for classifying the behavior of
individual cattle while the GNSS data is used to detect and track cattle herds
and their dispersion. In (Brennan et al., 2021), twenty accelerometry features
and three GNSS features are collectively used to distinguish between two basic
cattle behaviors of grazing and non-grazing.

In this work, we study making use of the complementary information available
through the accelerometry and GNSS data to enhance the accuracy of animal
behavior classification. We extract three new features from the GNSS data,
which relate to distance to the water point, Doppler-effect-borne speed, and
estimated horizontal position error. We develop two algorithms via two
different approaches for utilizing both accelerometry and GNSS features to
classify animal behavior. The first approach is based on the concatenation of
the accelerometry and GNSS features and using a multilayer perceptron (MLP)
classifier. The second approach is based on classifying animal behavior using
two MLP classifiers, each taking the features of one sensing mode, i.e.,
accelerometry or GNSS features, as input followed by fusing the posterior
class probabilities predicted by the two MLP classifiers and inferring the
behavior class from the fused posterior probabilities. This second approach
has been adopted in various machine learning applications involving the
combination of multiple classifiers (Kittler et al., 1998). However, to the
best of our knowledge, it has not been considered for animal behavior
classification using multimodal sensor data.

We use two datasets that contain concurrent accelerometry and GNSS data to
evaluate the performance of the developed multimodal animal behavior
classification algorithms. The datasets are collected via smart cattle collar
tags and ear tags during a field trail. We make these datasets publicly
available111https://github.com/Reza219/Animal_behavior_classification_Acc_GNSS.
The leave-one-animal-out cross-validation results show that the combined
utilization of the accelerometry and GNSS data offers improved classification
accuracy over using the data of only one sensing mode. In addition, we observe
that the algorithm based on the posterior probability fusion approach is more
accurate, efficient, and scalable compared with the one based on concatenating
the features despite its underlying independence assumption. The devised
algorithms can be extended to be used with different or higher number of
sensing modes. The advantages of the approach based on posterior probability
fusion are likely to be more pronounced when the number of sensing modes is
high and the statistical interdependence of the data coming from different
sensing modes given the target class is sufficiently low.

##  2 Calculation of features

In this section, we describe the features that we calculate from accelerometry
and GNSS data.

###  2.1 Features of accelerometry data

The features that we calculate from accelerometry data are similar to those
proposed in (Arablouei et al., 2021), i.e., the mean and mean-absolute-
filtered features. We describe their calculation in the following.

Let the $N$-dimensional vectors $\mathbf{a}_{ix}$, $\mathbf{a}_{iy}$, and
$\mathbf{a}_{iz}$ denote the triaxial accelerometer readings associated with
the $i$th datapoint ($i=1,2,\dots$) in three spatial axes, i.e., $x$, $y$, and
$z$, respectively. In this work, we consider $N=256$.

To calculate the mean features, we average the entries of $\mathbf{a}_{i}d$
for each axis $d\in\\{x,y,z\\}$ as

$m_{id}=\frac{1}{N}\mathbf{1}^{\intercal}\mathbf{a}_{id},\ d\in\\{x,y,z\\}$
(1)

where $\mathbf{1}$ is the all-ones column vector of appropriate size. These
features relate to the animal’s neck/head pose.

To calculate the next set of features, we pass the accelerometer readings of
each axis through a first-order high-pass Butterworth filter that has a single
parameter $\gamma_{d}$, $d\in\\{x,y,z\\}$. These infinite impulse response
(IIR) filters remove the effect of gravity. We express their implementation by

$\left[1,-\gamma_{d}\right]^{\intercal}*\tilde{\mathbf{a}}_{id}=\left[1,-1\right]^{\intercal}*\mathbf{a}_{id},\
d\in\\{x,y,z\\}$ (2)

where $\tilde{\mathbf{a}}_{id}$ is the filter output and $*$ denotes the
linear convolution operation. This notation is a time-domain representation of
the utilized IIR filters, which we implement recursively as described in
(Arablouei et al., 2021).

We average the absolute values of the high-pass-filtered accelerometer
readings for each axis to calculate the second set of features as

$s_{id}=\frac{1}{N}\mathbf{1}^{\intercal}|\tilde{\mathbf{a}}_{id}|,\
d\in\\{x,y,z\\}.$ (3)

These features relate to the intensity of the animal’s body movements.

We collect both sets of above-mentioned features, calculated from the
accelerometry data of the datapoint $i$, in the feature vector
$\mathbf{f}_{ia}\in\mathbb{R}^{6\times 1}$.

###  2.2 Features of GNSS data

We calculate three GNSS features for each datapoint indexed by $i$ using four
sets of values reported by the GNSS receiver. These values are latitude,
longitude, speed, and estimated horizontal position error (EHPE), which are
provided by the GNSS receiver during the time segment corresponding to each
datapoint. The GNSS receiver data is reported approximately every one second.
Therefore, during a typical time segment of five seconds, on average, five
sets of GNSS data are available. The EHPE is the radius of a circle around the
actual location that is estimated to contain the GNSS-estimated position with
the probability associated with one standard-deviation of uncertainty, which
is approximately 68% if we assume Gaussian distribution for the perturbation
causing the uncertainty. Here, we simply refer to EHPE as _error_.

We compute two features, denoted by $v_{i}$ and $e_{i}$, as the medians of the
speed and error values, respectively. The median speed feature, $v_{i}$, can
be linked to the approximate distance traveled by the animal during the time
window of each datapoint. Since GNSS speed is calculated using Doppler shift,
it provides a better estimate of traveled distance per unit time compared to
the distance between consecutive positions reported by the GNSS receiver. We
choose to compute the median values instead of the means to have features that
are more robust to outliers and erroneous readings.

We obtain another feature by calculating the distance of the water point from
the median position of the collar, whose coordinates are the medians of the
corresponding latitude and longitude values for each datapoint. We keep the
water point at a fixed location during every day of data collection
experiments and record its coordinates daily. To calculate the distance to the
water point (DtWP) feature, we use the Pythagorean theorem on an
equirectangular projection. This method is approximate. However, it is
sufficiently accurate for short distances while being more computationally
efficient compared to more accurate methods such as the haversine formula.
Hence, we compute the DtWP feature, denoted by $\delta_{i}$, as

$\displaystyle\delta_{i}$
$\displaystyle=r\sqrt{\cos^{2}\left(\frac{\tilde{\phi}_{i}+\phi_{w}}{2}\right)\left(\tilde{\lambda_{i}}-\lambda_{w}\right)^{2}+\left(\tilde{\phi_{i}}-\phi_{w}\right)^{2}}$
(4)

where $r$ is the radius of earth (approximately 6,371,230 meters), $\phi_{w}$
and $\lambda_{w}$ are, respectively, the known latitude and longitude of the
water point, and $\tilde{\phi_{i}}$ and $\tilde{\lambda_{i}}$ are the median
latitude and longitude values for the $i$th datapoint, respectively.

We stack the three features of speed, error, and DtWP calculated from the GNSS
data of the datapoint $i$ in the feature vector
$\mathbf{f}_{ig}\in\mathbb{R}^{3\times 1}$.

##  3 Classification of behavior

In this section, we describe two methods for classifying animal behavior
through combining the information coming from accelerometry and GNSS data.

###  3.1 Concatenation of features

The first method that we consider is to concatenate the feature vectors
corresponding to the accelerometry and GNSS data and classify the behavior
using the augmented feature vector, i.e.,
$\mathbf{f}_{i}=\left[\mathbf{f}_{ia}^{\intercal},\mathbf{f}_{ig}^{\intercal}\right]^{\intercal}$.

For the classification part, we feed the feature vector $\mathbf{f}_{i}$ into
an MLP classifier that has a single hidden layer and uses the rectified linear
unit (ReLU) activation function. We express the output of the MLP as

$\mathbf{z}_{i}=\mathbf{W}_{2}\max\left(\mathbf{0},\mathbf{W}_{1}\mathbf{f}_{i}+\mathbf{b}_{1}\right)+\mathbf{b}_{2}$
(5)

where $\mathbf{0}$ is the zero vector, $\mathbf{W}_{1}\in\mathbb{R}^{L\times
F}$ and $\mathbf{b}_{1}\in\mathbb{R}^{L\times 1}$ are the weight matrix and
the bias vector of the hidden layer, $\mathbf{W}_{2}\in\mathbb{R}^{C\times L}$
and $\mathbf{b}_{2}\in\mathbb{R}^{C\times 1}$ are the weight matrix and the
bias vector of the output layer, $F$ is the number of features, $L$ is the
dimension of the output of the hidden layer, and $C$ is the number of classes.

We use the $\mathrm{softmax}$ function to convert the output of the MLP to the
normalized scores, which represent the posterior probabilities of belonging to
the considered behavior classes, given the model is well calibrated.
Therefore, we calculate the posterior probabilities as

$p(y_{c}|\mathbf{f}_{i})=\frac{e^{z_{ic}}}{\mathbf{1}^{\intercal}e^{\mathbf{z}_{i}}},\
c=1,\dots,C$ (6)

where $y_{c}$ is the binary variable that indicates association to class
$c\in\\{1,\dots,C\\}$, $z_{ic}$ is the $c$th entry of $\mathbf{z}_{i}$, and
the exponentiation in the denominator is applied element-wise.

At training time, we require the normalized scores or the posterior
probabilities $p(y_{c}|\mathbf{f}_{i})$ to calculate the cross-entropy loss
associated with datapoint $i$. However, at inference time, to save on
computations, we determine the inferred behavior class by finding the largest
corresponding MLP output without applying the $\mathrm{softmax}$ function,
i.e., by calculating $\arg\max_{c}{z_{ic}}$.

We depict the block diagram of the above-described method based on feature
concatenation in Fig. 1.

Figure 1: The block diagram of the animal behavior classification algorithm
using multimodal sensor data via the feature concatenation method. Figure 2:
The block diagram of the animal behavior classification algorithm using
multimodal sensor data via the posterior probability fusion method.

###  3.2 Fusion of posterior probabilities

The second method that we consider is based on fusing the posterior
probabilities predicted by two MLP classifiers that are separately fed with
the features calculated from the accelerometry and GNSS data as shown in Fig.
2. Each MLP classifier has one hidden layer and both MLPs use the ReLU
activation function. The outputs of the MLPs corresponding to the
accelerometry and GNSS data, respectively, are

$\displaystyle\mathbf{z}_{ia}$
$\displaystyle=\mathbf{W}_{2a}\max\left(\mathbf{0},\mathbf{W}_{1a}\mathbf{f}_{ia}+\mathbf{b}_{1a}\right)+\mathbf{b}_{2a}$
(7) $\displaystyle\mathbf{z}_{ig}$
$\displaystyle=\mathbf{W}_{2g}\max\left(\mathbf{0},\mathbf{W}_{1g}\mathbf{f}_{ig}+\mathbf{b}_{1a}\right)+\mathbf{b}_{2g}$
(8)

where $\mathbf{W}_{1a}$, $\mathbf{W}_{2a}$, $\mathbf{W}_{1g}$, and
$\mathbf{W}_{2g}$ are the relevant weight matrices and $\mathbf{b}_{1a}$,
$\mathbf{b}_{2a}$, $\mathbf{b}_{1g}$, and $\mathbf{b}_{2g}$ are the related
bias vectors. Applying the $\mathrm{softmax}$ function to the MLP outputs
yields the corresponding posterior probabilities as

$\displaystyle p(y_{c}|\mathbf{f}_{ia})$
$\displaystyle=\frac{e^{z_{iac}}}{\mathbf{1}^{\intercal}e^{\mathbf{z}_{ia}}},\
c=1,\dots,C$ (9) $\displaystyle p(y_{c}|\mathbf{f}_{ig})$
$\displaystyle=\frac{e^{z_{igc}}}{\mathbf{1}^{\intercal}e^{\mathbf{z}_{ig}}},\
c=1,\dots,C$ (10)

where $z_{iac}$ and $z_{igc}$ are the $c$th entries of $\mathbf{z}_{ia}$ and
$\mathbf{z}_{ig}$, respectively.

Here, our objective is to compute the posterior probability for each behavior
class given the feature vectors $\mathbf{f}_{ia}$ and $\mathbf{f}_{ig}$, i.e.,
$p(y_{c}|\mathbf{f}_{i})$, $c\in\\{1,\dots,C\\}$, using the sensor-specific
posterior probabilities, i.e., $p(y_{c}|\mathbf{f}_{ia})$ and
$p(y_{c}|\mathbf{f}_{ig})$, $c\in\\{1,\dots,C\\}$, produced by the respective
classifiers and the prior probability of each behavior class, i.e.,
$p(y_{c})$, $c\in\\{1,\dots,C\\}$.

_Assumption_ : To make the fusion of the posterior probabilities tractable, we
assume that the likelihoods of observing $\mathbf{f}_{ia}$ and
$\mathbf{f}_{ig}$ are conditionally independent of each other given the
behavior class $y_{c}$ for every class and datapoint, i.e.,
$p\left(\mathbf{f}_{ia}|y_{c}\right)$ and
$p\left(\mathbf{f}_{ig}|y_{c}\right)$ $\forall i\mathrm{\ and\
}c\in\\{1,\dots,C\\}$ are statistically independent. Therefore, we have

$p\left(\mathbf{f}_{i}|y_{c}\right)=p\left(\mathbf{f}_{ia}|y_{c}\right)p\left(\mathbf{f}_{ig}|y_{c}\right),\
\forall i\mathrm{\ and\ }c\in\\{1,\dots,C\\}.$ (11)

We also make the implicit assumption that the behavior classes are mutually
exclusive that means only one behavior class can be associated with each
feature vector.

Using the Bayes theorem and the above independence assumption, we have

$\displaystyle p\left(y_{c}|\mathbf{f}_{i}\right)$
$\displaystyle=\frac{p\left(\mathbf{f}_{i}|y_{c}\right)p\left(y_{c}\right)}{p\left(\mathbf{f}_{i}\right)}$
(12)
$\displaystyle=\frac{p\left(\mathbf{f}_{ia}|y_{c}\right)p\left(\mathbf{f}_{ig}|y_{c}\right)p\left(y_{c}\right)}{p\left(\mathbf{f}_{i}\right)}$
(13)
$\displaystyle=\frac{p\left(\mathbf{f}_{ia}|y_{c}\right)p\left(y\right)p\left(\mathbf{f}_{ig}|y_{c}\right)p\left(y\right)}{p\left(\mathbf{f}_{i}\right)p\left(y\right)}$
(14)
$\displaystyle=\frac{p\left(\mathbf{f}_{ia}\right)p\left(\mathbf{f}_{ig}\right)}{p\left(\mathbf{f}_{i}\right)}\times\frac{p\left(y_{c}|\mathbf{f}_{ia}\right)p\left(y_{c}|\mathbf{f}_{ig}\right)}{p\left(y_{c}\right)}$
(15)
$\displaystyle\propto\frac{p\left(y_{c}|\mathbf{f}_{ia}\right)p\left(y_{c}|\mathbf{f}_{ig}\right)}{p\left(y_{c}\right)}$
(16)

where $\propto$ denotes proportionality. The first multiplicand in (15) is
independent of $y_{c}$ while the values of the features are known. Therefore,
its value is the same for all classes making it irrelevant to classification.
Consequently, to find the class with the highest fused posterior probability,
we only need to find the class with the highest value of the right-hand side
of (16). Nonetheless, we can calculate the fused posterior probabilities for
each class without the knowledge of the statistical distribution of the
features via

$\displaystyle q_{c}\left(\mathbf{f}_{i}\right)$
$\displaystyle=\frac{p\left(y_{c}|\mathbf{f}_{ia}\right)p\left(y_{c}|\mathbf{f}_{ig}\right)}{p\left(y_{c}\right)}$
(17) $\displaystyle p\left(y_{c}|\mathbf{f}_{i}\right)$
$\displaystyle=\frac{q_{c}\left(\mathbf{f}_{i}\right)}{\sum_{c^{\prime}=1}^{C}{q_{c^{\prime}}\left(\mathbf{f}_{i}\right)}},\
c\in\\{1,\dots,C\\}.$ (18)

Taking the logarithm of both sides of (16) and using (9) and (10), we have

$\displaystyle\ln p\left(y_{c}|\mathbf{f}_{i}\right)$
$\displaystyle\propto\ln\frac{e^{z_{iac}}e^{z_{igc}}}{p\left(y_{c}\right)\mathbf{1}^{\intercal}e^{\mathbf{z}_{ia}}\mathbf{1}^{\intercal}e^{\mathbf{z}_{ig}}}$
(19) $\displaystyle\propto
z_{iac}+z_{igc}-\ln{p(y_{c})}-\ln\left(\mathbf{1}^{\intercal}e^{\mathbf{z}_{ia}}\mathbf{1}^{\intercal}e^{\mathbf{z}_{ig}}\right)$
(20) $\displaystyle\propto z_{iac}+z_{igc}-\ln{p(y_{c})}.$ (21)

Thus, we have

$\arg\max_{c}{p\left(y_{c}|\mathbf{f}_{i}\right)}=\arg\max_{c}{\left(z_{iac}+z_{igc}-\ln{p(y_{c})}\right)},$
(22)

which means, to determine the class with the highest fused posterior
probability efficiently, we can calculate only the values of
$z_{iac}+z_{igc}-\ln{p(y_{c})}$, $c\in\\{1,\dots,C\\}$, and find the class
with the largest corresponding value.

##  4 Datasets

In this section, we describe two datasets containing both accelerometry and
GNSS data that we use for evaluating the proposed multimodal animal behavior
classification algorithms. We also explain the associated data collection
experiment and annotation procedure.

###  4.1 Data collection

To evaluate the algorithms described in sections 2 and 3, we use two datasets.
We have obtained these datasets by running a field experiment for eight days
during 18 to 27 March 2020 at the Commonwealth Scientific and Industrial
Research Organisation (CSIRO) FD McMaster Laboratory Pasture Intake Facility
(Greenwood et al., 2014), Chiswick NSW, Australia ($30\degree
36^{\prime}28.17^{\prime\prime}$S, $151\degree
32^{\prime}39.12^{\prime\prime}$E). The experiment was approved by the CSIRO
FD McMaster Laboratory Chiswick Animal Ethics Committee with the animal
research authority number 19/18.

We collected tri-axial accelerometry and GNSS data from eight Angus heifers
using eGrazor collar
tags222https://www.csiro.au/en/research/animals/livestock/egrazor-measuring-
cattle-pasture-intake and Ceres Tag ear tags333https://www.cerestag.com/. All
eight cattle were fitted with collar tags and six also wore ear tags. The
experiment paddock consisted of four plots of approximately 25 x 100 m that
were fully enclosed with electric fences. During each day of the experiment,
the cattle were confined to an area of approximately 25 x 25 m. This way, the
animals were in the field of view of the GoPro Hero 5
cameras444https://gopro.com/, which we placed at four corners of the
experiment area and used to record the behavior of the animals during the
experiment with a resolution of $1920\times 1080$ pixels and 30 frames per
second.

Each eGrazor collar tag houses a 32GB SD flash memory card. However, the on-
board flash memory of the Ceres Tag ear tags is limited to 1MB. Therefore,
during the experiment, we logged the accelerometry and GNSS data of both
collar tags and ear tags on the SD cards of the collar tags. To this end, we
streamed the accelerometry and GNSS data of each ear tag to the collar tag
worn by the same animal via a generic attribute profile (GATT) Bluetooth link.
At the end of the experiment, we collected the SD cards and retrieved the
logged data.

Fig. 3 is a satellite view of the experiment site where we have marked the
fence lines of the utilized plots with dashed lines. The figure also includes
the location of the pen that we used for installing and removing the collar
and ear tags as well as keeping the cattle at night. Fig. 4 is a snapshot of
the paddock and the cattle used in the experiment wearing collar and ear tags.
An overview of the hardware specifications for the eGrazor collar tag and
Ceres Tag ear tag can be found in (Arablouei et al., 2022) and (Wang et al.,
2021), respectively.

The IMU used in the collar tags is InvenSense
MPU-9250555https://invensense.tdk.com/products/motion-
tracking/9-axis/mpu-9250/ and the accelerometer used in the ear tags is Bosch
Sensortec BMA400666https://www.bosch-sensortec.com/products/motion-
sensors/accelerometers/bma400/. We set the accelerometer sample rate to $50$
Hz on the collar tags and to $62.5$ Hz on the ear tags. Therefore, the time
window corresponding to $256$ consecutive accelerometer samples is about
$5.12$ s for the collar tag accelerometry data and about $4.1$ s for the ear
tag accelerometry data.

The GNSS receiver used in the collar tags is u-blox
MAX-M8C777https://www.u-blox.com/en/product/max-m8-series and the one used in
the ear tags is u-blox
ZOE-M8888https://www.u-blox.com/en/product/zoe-m8-series. We logged the data
reported by the GNSS receivers of the collar tags with the frequency of
approximately once per second. Every ear tag also features a GNSS receiver.
However, it is intended to be used with much lower frequency due to the small
capacity of the ear tag’s battery and photovoltaic module (solar panel).
Therefore, the GNSS data that we use in this work came from the collar tags.

Figure 3: The satellite view of the experiment site including the fence lines
of the utilized plots and the location of the pen where cattle were kept at
night. Figure 4: Cattle on the paddock wearing collar and ear tags during the
data collection experiment.

###  4.2 Annotations

We annotated sections of the collected accelerometry and GNSS data by
observing the behavior of the cattle in the videos recorded via the GoPro
cameras during the experiment. Experts logged the annotation instances using a
mobile application specifically built for this purpose. We used the
annotations to generate our labeled datasets called Arm20c and Arm20e, which
correspond to the collar and ear tag data, respectively.

We consider five mutually-exclusive cattle behaviors of grazing, walking,
resting, drinking, and alia. The resting behavior class also includes the
ruminating behavior. The alia class encompasses all behaviors other than
grazing, walking, resting, and drinking. To avoid confusion, we use the Latin
word alia instead of other, which we have previously used for the same
purpose.

We first use the annotations to label the corresponding accelerometry data.
Afterwards, we create the datapoints of each dataset by dividing the related
labeled tri-axial accelerometry data into non-overlapping segments of $256$
consecutive readings. We then find the GNSS receiver data whose timestamps
fall into the time window of every segment in each dataset and associate them
with the corresponding labeled accelerometry data to form the datapoints
containing both accelerometry and GNSS data. We ignore the instances when
either the accelerometry or GNSS data is missing. The median number of the
GNSS readings of the datapoints is five in the Arm20c dataset and four in the
Arm20e dataset. In Table 1, we give the number of labeled datapoints for each
behavior class in each dataset.

Table 1: The number of labeled datapoints for each behavior class in the considered datasets. behavior dataset | Arm20c | Arm20e
---|---|---
grazing | 6156 | 5833
walking | 910 | 854
resting | 4080 | 3491
drinking | 594 | 501
alia | 222 | 200
total | 11962 | 10879

##  5 Results

In this section, we evaluate the performance of the proposed multimodal
behavior classification algorithms in terms of classification accuracy and
computational complexity using the datasets described in section 4. We also
provide some visual analysis of the labeled GNSS data as well as the feature
spaces associated with both accelerometry and GNSS data.

We make the utilized datasets as well as the Python code for reproducing the
presented results publicly
available999https://github.com/Reza219/Animal_behavior_classification_Acc_GNSS.

###  5.1 Visualization of GNSS data

In Fig. 5, we plot the GNSS-estimated positions (longitude and latitude
coordinates) of all datapoints for the Arm20c dataset. We plot the estimated
positions in separate figures for each calendar day and color them according
to their corresponding behavior class label. During every day of the data
collection experiment, we changed the location of the water point and used
different 25m-by-25m sections of the paddocks to maintain the cattle within
the field of view of the cameras utilized to record the footage of the
experiment. Therefore, in Fig. 5, we also indicate the location of the water
point and those of the cameras installed at the corners of each daily
experiment area as well as the permanent and temporary fence lines outlining
the area used for the experiment each day.

(a) 23/March/2020

(b) 24/March/2020

(c) 25/March/2020

(d) 26/March/2020

(e) 27/March/2020

Figure 5: The GNSS-estimated position coordinates of the datapoints in the
Arm20c dataset for every calendar day of the experiment leading to the
dataset. Each position point is colored according to its associated behavior
class.

In Fig. 6, we plot the GNSS-estimated positions for all datapoints of the
Arm20c dataset and the entire duration of the experiment. For clarity, we plot
the estimated positions in three different figures corresponding to the
grazing behavior, the resting behavior, and the remaining rare behaviors,
i.e., walking, drinking, and alia.

(a) grazing

(b) resting

(c) walking, drinking, and alia

Figure 6: The GNSS-estimated position coordinates of the datapoints in the
Arm20c dataset. Each position point is colored according to its associated
behavior class.

As evident from Figs. 5 and 6 in agreement with intuition, the positions of
the animals alone may not carry significant information about their behavior
as they exhibit several behaviors such as grazing, resting, walking, and
grooming at various locations on the paddock. However, the animals’ distance
from the places of interest such as the water point is more likely to contain
discriminative information beneficial for behavior classification. It is clear
from Fig. 6(c) that the animals are in the proximity of the water point
wherever their behavior is annotated as drinking. Note that the positions at
which the animals’ behavior is annotated as walking include the paths that
connect the cattle pen to the daily experiment areas. The paths are distinctly
visible in Figs. 5 and 6(c).

To gain some insights into the values of the features calculated from the GNSS
data and their correlation with the behavior class, in Fig. 7, we plot the
GNSS-estimated positions and color them according to their corresponding
values of the GNSS features, i.e., DtWP, speed, and error, for all datapoints
of the Arm20c dataset. In addition, we plot the histograms of the values of
the GNSS features for each behavior class in Fig. 8. The histograms are also
related to the Arm20c dataset. However, those for the Arm20e dataset are
qualitatively similar.

(a) DtWP

(b) speed

(c) error

Figure 7: The GNSS-estimated position coordinates of the datapoints in the
Arm20c dataset colored according to their corresponding values of DtWP, speed,
and error.

(a) DtWP

(b) speed

(c) error

Figure 8: The histograms of the values of the GNSS features for the Arm20c
dataset and each behavior class.

Figs. 7(a) and 8(a) show that the values of the DtWP feature are lower when
the animals drink from the water point compared to when they exhibit other
behaviors. It is also evident in Figs. 7(b) and 8(b) that the speed feature
often has significantly higher values when the animals walk compared to when
they perform other behaviors. Therefore, we can expect that the addition of
these features to the features extracted from the accelerometry data should
help improve the accuracy of behavior classification, especially regarding the
uncommon behavior classes of walking and drinking. On the other hand, the
values of the error feature do not appear to correlate with the behavior in
any discernible manner, hence not carry any substantial information about the
behavior directly. However, as we will show in section 5.3, the error feature
can be informative for behavior classification, specifically, when used in
conjunction with the other GNSS features.

###  5.2 Classification performance

We evaluate the classification accuracy of the considered algorithms and tune
their hyperparameters via leave-one-animal-out cross-validation using our
labeled datasets Arm20c and Arm20e. We calculate the cross-validated results
by aggregating the results of all cross-validation folds where, in each fold,
we use the datapoints associated with one animal for validation and the rest
for training.

We use the Matthews correlation coefficient (MCC) (Matthews, 1975) for
evaluating the classification accuracy. MCC takes into account true and false
positives and negatives and is meaningful even with highly imbalanced
datasets. It has values between $-1$ and $+1$ where $+1$ is perfect
prediction, $0$ is no better than random prediction, and $-1$ is perfect
inverse prediction.

With the Arm20c dataset, we use six accelerometry features as described in
section 2.1 while setting the values of the filter parameters to
$\gamma_{d}=0.75$, $d\in\\{x,y,z\\}$, which we determine empirically through
cross-validation. With the Arm20e dataset, we use nine accelerometry features,
six of which are computed in the same way as in section 2.1 while setting
$\gamma_{d}=0.75$, $d\in\\{x,y,z\\}$. The other three features are an extra
set of $s_{id}$, $d\in\\{x,y,z\\}$, features with the associated filter
parameter values of $\gamma_{d}=0.5$, $d\in\\{x,y,z\\}$. These extra features
provide more information about the intensity of the animal’s body motion,
particularly, at the higher parts of the associated frequency spectrum as with
$\gamma_{d}=0.5$ lower frequencies are filtered out more aggressively compared
to with $\gamma_{d}=0.75$. This can be noted in (Arablouei et al., 2021, Fig.
4). The supplementary information provided by these additional features helps
improve the classification accuracy with the Arm20e dataset.

We implement the MLP classifiers using the scikit-learn Python
package101010https://scikit-learn.org/. We utilize the L-BFGS algorithm (Liu
and Nocedal, 1989) to optimize the cross-entropy loss associated with the
classification. We set the number of units in the hidden layer of each MLP to
the average of the numbers of its input and output neurons, which are the
corresponding features and classes, rounded upward. We tune the
$\ell_{2}$-norm regularization hyperparameter of each MLP for each dataset in
conjunction with our leave-one-animal-out cross-validation process. To train
each MLP, we run the L-BFGS optimization algorithm for maximum $10,000$
iterations.

We initialize the model parameters (MLP weights and biases) of each classifier
randomly. As the classification accuracy may vary depending on the initial
values of the parameters, we repeat each cross-validation evaluation $100$
times using independently selected random initial parameter values and compute
the average and standard deviation of the MCC values over the trials.
Therefore, when reporting MCC values, we present the mean over $100$
independent runs followed by the associated standard-deviation separated by
the $\pm$ sign.

In Tables 2 and 3, we present the cross-validated classification accuracy
results, in terms of MCC, for the Arm20c and Arm20e datasets, respectively.
The tables include the MCC values for each behavior class as well as the
overall MCC values. The results are for the classifier that only uses the GNSS
features (GNSS), the classifier that only uses the accelerometry features
(Acc), the classifier that uses the concatenated GNSS and accelerometry
features (FC), and the classifier that fuses the posterior class probabilities
predicted by the GNSS and Acc classifiers (PF).

Table 2: The leave-one-animal-out cross-validated MCC values of the considered
algorithms for the Arm20c dataset and each behavior class. The MCC values are
averaged over $100$ independent runs with random initialization.
behavior algorithm | GNSS | Acc | FC | PF
---|---|---|---|---
grazing | $0.4348\pm 0.0070$ | $0.9011\pm 0.0031$ | $0.9252\pm 0.0040$ | $0.9227\pm 0.0037$
walking | $0.7708\pm 0.0130$ | $0.8361\pm 0.0048$ | $0.8940\pm 0.0101$ | $0.8900\pm 0.0131$
resting | $0.3931\pm 0.0081$ | $0.9000\pm 0.0063$ | $0.8956\pm 0.0059$ | $0.9050\pm 0.0046$
drinking | $0.1952\pm 0.0148$ | $0.6444\pm 0.0186$ | $0.7508\pm 0.0172$ | $0.7679\pm 0.0114$
alia | $-$ | $0.2785\pm 0.0337$ | $0.3755\pm 0.0292$ | $0.3256\pm 0.0319$
overall | $0.4375\pm 0.0063$ | $0.8594\pm 0.0045$ | $0.8829\pm 0.0048$ | $0.8847\pm 0.0043$
Table 3: The leave-one-animal-out cross-validated MCC values of the considered
algorithms for the Arm20e dataset and each behavior class. The MCC values are
averaged over $100$ independent runs with random initialization.
behavior algorithm | GNSS | Acc | FC | PF
---|---|---|---|---
grazing | $0.3516\pm 0.0063$ | $0.7659\pm 0.0070$ | $0.8122\pm 0.0060$ | $0.8140\pm 0.0055$
walking | $0.7545\pm 0.0118$ | $0.6607\pm 0.0155$ | $0.8204\pm 0.0135$ | $0.8305\pm 0.0098$
resting | $0.3138\pm 0.0084$ | $0.7229\pm 0.0069$ | $0.7431\pm 0.0074$ | $0.7494\pm 0.0065$
drinking | $0.1179\pm 0.0201$ | $0.0841\pm 0.0287$ | $0.4352\pm 0.0185$ | $0.4487\pm 0.0227$
alia | $-$ | $0.1191\pm 0.0444$ | $0.1717\pm 0.0370$ | $0.1192\pm 0.0296$
overall | $0.3713\pm 0.0057$ | $0.6916\pm 0.0064$ | $0.7475\pm 0.0059$ | $0.7532\pm 0.0051$

The results in Tables 2 and 3 show that combining the GNSS and accelerometry
features improves the classification accuracy appreciably while fusing the
posterior probabilities produced by the classifiers that use GNSS and
accelerometry features separately appears to outperform the classifier that
uses the concatenated features. We also observe that the performance
enhancement offered by the GNSS features is more prominent for the uncommon
behaviors of walking and drinking, especially with the Arm20e dataset. This is
not surprising since, as discussed in section 5.1, the DtWP and speed features
are expected to provide additional information that can help better
distinguish the drinking and walking instances. Note that using the GNSS
features alone does not lead to any meaningful classification of the drinking
behavior. However, when these features are used in conjunction with the
accelerometry features, the accuracy of classifying the drinking behavior
improves substantially.

###  5.3 Ablation study

In Tables 4 and 5, we provide the cross-validated MCC values of the PF
algorithm when different combinations of the GNSS features are used alongside
the accelerometry features for both considered datasets and each behavior
class. It is clear from the results that, among the considered GNSS features,
DtWP is the most informative one regarding the classification of all
considered behaviors. The other two GNSS features, speed and error, are
beneficial to behavior classification with Arm20e dataset. However, they do
not seem to offer any noticeable advantage with the Arm20c dataset. This may
be because of higher complexity of classifying animal behavior using the ear
tag accelerometry data that makes it more amenable to exploit any extra
knowledge imparted by additional features such as speed and error. Moreover,
the error feature appears to be more helpful when it is used together with the
DtWP feature. This is possibly due to the error feature carrying some
information about how reliable the DtWP feature value of each datapoint is.

The results of Tables 4 and 5 show that the DtWP feature helps improve the
accuracy of classifying the drinking behavior significantly with both
considered datasets. It also helps with the classification of the walking
behavior in both datasets. In addition, using the speed feature enhances the
classification accuracy of the walking behavior with the Arm20e dataset. All
considered GNSS features can benefit the classification of the grazing and
resting behaviors as well, although to a lesser extent compared to the
drinking and walking behaviors and to varying degrees given different
datasets.

Table 4: The leave-one-animal-out cross-validated MCC values of the PF
algorithm when using different combinations of the GNSS features in the GNSS-
based classifier for the Arm20c dataset and each behavior class.
behavior GNSS features | all |  | DtWP
---
error
| DtWP
---
speed
| speed
---
error
DtWP | speed | error | none
grazing | $0.9227\pm 0.0037$ | $0.9222\pm 0.0031$ | $0.9197\pm 0.0049$ | $0.9038\pm 0.0033$ | $0.9216\pm 0.0040$ | $0.8938\pm 0.0036$ | $0.9033\pm 0.0038$ | $0.9011\pm 0.0031$
walking | $0.8900\pm 0.0131$ | $0.8898\pm 0.0044$ | $0.8891\pm 0.0145$ | $0.8453\pm 0.0100$ | $0.8889\pm 0.0035$ | $0.8391\pm 0.0086$ | $0.8196\pm 0.0061$ | $0.8361\pm 0.0048$
resting | $0.9050\pm 0.0046$ | $0.9041\pm 0.0050$ | $0.9022\pm 0.0064$ | $0.8950\pm 0.0076$ | $0.9040\pm 0.0050$ | $0.8982\pm 0.0072$ | $0.8968\pm 0.0074$ | $0.9000\pm 0.0063$
drinking | $0.7679\pm 0.0114$ | $0.7664\pm 0.0129$ | $0.7623\pm 0.0156$ | $0.6442\pm 0.0243$ | $0.7674\pm 0.0125$ | $0.6453\pm 0.0190$ | $0.6460\pm 0.0230$ | $0.6444\pm 0.0186$
alia | $0.3256\pm 0.0319$ | $0.3096\pm 0.0331$ | $0.3045\pm 0.0272$ | $0.3142\pm 0.0292$ | $0.2895\pm 0.0267$ | $0.2980\pm 0.0291$ | $0.2696\pm 0.0346$ | $0.2785\pm 0.0337$
overall | $0.8847\pm 0.0043$ | $0.8845\pm 0.0037$ | $0.8841\pm 0.0053$ | $0.8593\pm 0.0058$ | $0.8819\pm 0.0038$ | $0.8597\pm 0.0055$ | $0.8587\pm 0.0056$ | $0.8594\pm 0.0045$
Table 5: The leave-one-animal-out cross-validated MCC values of the PF
algorithm when using different combinations of the GNSS features in the GNSS-
based classifier for the Arm20e dataset and each behavior class.
behavior GNSS features | all |  | DtWP
---
error
| DtWP
---
speed
| speed
---
error
DtWP | speed | error | none
grazing | $0.8140\pm 0.0055$ | $0.8104\pm 0.0063$ | $0.7982\pm 0.0059$ | $0.7956\pm 0.0055$ | $0.7984\pm 0.0077$ | $0.7739\pm 0.0071$ | $0.7884\pm 0.0060$ | $0.7659\pm 0.0070$
walking | $0.8305\pm 0.0098$ | $0.8062\pm 0.0084$ | $0.7957\pm 0.0094$ | $0.7702\pm 0.0091$ | $0.7947\pm 0.0097$ | $0.7195\pm 0.0080$ | $0.6885\pm 0.0112$ | $0.6607\pm 0.0155$
resting | $0.7494\pm 0.0065$ | $0.7373\pm 0.0074$ | $0.7449\pm 0.0069$ | $0.7373\pm 0.0095$ | $0.7333\pm 0.0072$ | $0.7345\pm 0.0099$ | $0.7243\pm 0.0085$ | $0.7229\pm 0.0069$
drinking | $0.4487\pm 0.0227$ | $0.4444\pm 0.0268$ | $0.4572\pm 0.0226$ | $0.1149\pm 0.0350$ | $0.4546\pm 0.0239$ | $0.0938\pm 0.0318$ | $0.0940\pm 0.0333$ | $0.0841\pm 0.0287$
alia | $0.1192\pm 0.0296$ | $0.1072\pm 0.0372$ | $0.1029\pm 0.0266$ | $0.1152\pm 0.0379$ | $0.0990\pm 0.0359$ | $0.1166\pm 0.0368$ | $0.1189\pm 0.0423$ | $0.1191\pm 0.0444$
overall | $0.7532\pm 0.0051$ | $0.7443\pm 0.0058$ | $0.7411\pm 0.0053$ | $0.7228\pm 0.0069$ | $0.7371\pm 0.0063$ | $0.7058\pm 0.0072$ | $0.7042\pm 0.0061$ | $0.6916\pm 0.0064$

###  5.4 Computational complexity

Table 6 shows the number of parameters of the considered animal behavior
classification algorithms as well as the number of different arithmetic
operations required by their associated MLP classifier(s). The numbers include
the additional parameters and operations required by the PF algorithm for
fusing the posterior probabilities but exclude the parameters and operations
required for feature extraction as they are the same for the FC and PF
algorithms. As we see in Table 6, the PF algorithm requires fewer parameters
and arithmetic operations compared with the FC algorithm. The savings offered
by the PF algorithm over the FC algorithm are higher with the Arm20e dataset
as we extract more accelerometry features from this dataset.

Table 6: The number of parameters and the number of arithmetic operations
required by the classification part of the considered animal behavior
classification algorithms for both considered datasets.
|  | Arm20c | Arm20e
---|---|---|---
complexity dataset algorithm | GNSS | Acc | FC | PF | Acc | FC | PF
parameters | $32$ | $55$ | $98$ | $92$ | $98$ | $153$ | $135$
multiplications | $32$ | $55$ | $98$ | $87$ | $98$ | $153$ | $130$
additions/subtractions | $32$ | $55$ | $98$ | $97$ | $98$ | $153$ | $140$
ReLU operations | $4$ | $5$ | $7$ | $9$ | $7$ | $9$ | $11$
sum of operations | $68$ | $115$ | $203$ | $193$ | $203$ | $315$ | $281$

###  5.5 Feature-space visualization

To provide some insights into the benefits of combining the information
available from the GNSS and accelerometry data, we visualize the feature
subspace related to the Arm20c and Arm20e datasets using the $t$-distributed
stochastic neighbor embedding (tSNE) algorithm (van der Maaten and Hinton,
2008) in Figs. 9 and 10, respectively. The figures include the visualizations
of the feature subspace of each dataset when using the accelerometry features,
GNSS features, or both. Each point represents a datapoint and is colored
according to its corresponding behavior class. We calculate the features for
the entire datasets using the parameters of the respective models trained for
behavior classification. The tSNE algorithm preserves the local structure of
the feature subspace while projecting it onto a lower-dimensional space with
no guarantee on preserving the global structure of the data.

(a) Accelerometry features.

(b) GNSS features.

(c) Accelerometry and GNSS features.

Figure 9: The tSNE visualization of the feature subspace of the Arm20c dataset
using the accelerometry features, GNSS features, or both.

(a) Accelerometry features.

(b) GNSS features.

(c) Accelerometry and GNSS features.

Figure 10: The tSNE visualization of the feature subspace of the Arm20e
dataset using the accelerometry features, GNSS features, or both.

It is clear from Figs. 9 and 10 that augmenting the accelerometry features
with the GNSS features enhances the clustering of datapoints belonging to the
same class thus helps improve the classification accuracy. This is more
noticeable for the less common behavior classes, i.e., walking and drinking.

##  6 Discussion

In this work, we consider two sensing modes of accelerometry and GNSS.
However, our arguments and findings can be extended to any number or type of
sensing modes. In addition, we use MLP classifiers while any other
classification algorithm can also be used in principle. The only caveat is
that, for the PF algorithm to be effective, the classifiers associated to
individual sensing modes should either directly output class probabilities or
output quantities that can be converted to posterior class probabilities. When
using classifiers that do not intrinsically predict any probability
distribution over classes, such as the support vector machine, one can use a
calibration algorithm, such as Platt scaling (Platt, 1999), to transform the
classification model output to posterior probabilities.

The assumption of the accelerometry and GNSS features being conditionally
independent given the class label is akin to the underpinning assumption of
the naive Bayes algorithm, albeit when the features are treated in blocks.
However, the main difference between the PF algorithm and the naive Bayes
algorithm is that the former combines the posterior probabilities while the
latter combines the likelihoods.

Although the independence assumption may not be ideal, the PF algorithm has
several advantages over the FC algorithm that make it more useful in practice.
Particularly, statistical decoupling of the accelerometry and GNSS features
means that each classifier can operate independently in a lower-dimensional
parameter space. As a result, multimodal classification can be delegated to
smaller single-modal classifiers reducing overall computational and memory
requirements. This can also help mitigate problems that may arise with high-
dimensional data such as the need for datasets that scale exponentially with
the number of features.

Another important advantage of the PF algorithm over the FC algorithm is its
ability to effectively handle instances when either of the accelerometry or
GNSS data is corrupt or missing due to a fault or difference in the
availability frequency. While the FC algorithm fails in such instances, the PF
algorithm can still function and classify the behavior, although with reduced
accuracy. This can be realized by detecting the missing/corrupt sensor data
and outputting the predictions made by the classifier associated with the
valid sensor data instead of preforming the posterior probability fusion. In
the case of having more than two sensing modes, the contribution of the
invalid sensor data can be ignored in the probability fusion stage. On the
other hand, the FC algorithm may fail to make any meaningful prediction when
any feature is missing or corrupt as it requires all feature values to
estimate the posterior probabilities.

Fusing posterior probabilities predicted by multiple classifiers is most
sensible when the constituent classifiers are appropriately calibrated. In our
observation, both classifiers based on accelerometry and GNSS data are
sufficiently calibrated. However, even if the PF algorithm does not yield an
accurate estimate of the class probabilities, it makes the correct
classification decision as long as the target class is more probable than the
other ones. This holds even when the probability estimates are grossly
inaccurate, which is inconsequential in many applications. Therefore, the
underlying approximate probability model of the PF algorithm does not
necessarily hurt its robustness.

A minor disadvantage of the PF algorithm with respect to the FC algorithm is
that it requires the prior occurrence probabilities of the classes. This does
not necessarily pose any challenge or excessive burden as the prior class
probabilities can be easily calculated for any dataset during training. It is
important to ensure that this prior knowledge is appropriate for the real-
world settings in which the animal behavior classification algorithm is
deployed.

The computational complexity and memory requirement of the examined FC and PF
algorithms are only moderately higher compared with those of the
classification algorithm based on the accelerometry data alone, i.e. the
algorithm proposed in (Arablouei et al., 2021). Therefore, both FC and PF
algorithms run seamlessly on the embedded systems of our collar and ear tags
and fulfill in-situ classification of animal behavior.

Despite their usefulness and essential role in evaluating the performance of
different algorithms considered in this work, our datasets have some
limitations. First, the data collection took place in a rather small area of
25 x 25 m, albeit alternated daily. Such dimensions do not correspond to the
true conditions that most grazing beef cattle experience during most of their
lives. However, the confinement was required to make the behavior of the
cattle visible through the cameras placed at the corners of the experiment
area. This was in turn to enable accurate post-hoc annotations via inspecting
the recorded footage as well as collecting high-quality video data for other
research activities. As a result, the cattle did not require nor had much
opportunity to walk within the paddock. Hence, most of our data labeled as
walking relates to the occasions when the cattle were driven from the
overnight cattle pen to the experiment paddock and back. This has led to a
sizable proportion of walking instances to occur far from the water point. An
unnatural consequence of this is that the DtWP feature carries significant
information about the walking behavior compared to the speed feature. This
might have hindered true evaluation of the use of GNSS speed data for
improving the classification accuracy of walking behavior in grazing cattle.

Another detriment of the small experiment areas used for the data collection
is that the average GNSS position estimation error of approximately seven
meters is substantial relative to the area the cattle were contained within.
This limits the utility of the DtWP feature. In our future field trials, we
plan to employ differential or real-time kinematics GNSS positioning for
enhanced precision and expand the experiment areas as much as practical.

Classifying rare or infrequent animal behaviors such as the drinking behavior
of grazing cattle is fundamentally challenging. This is primarily due to the
paucity of training data that can practically be obtained for such behaviors.
Annotating the rare behaviors is also difficult since they happen occasionally
and in short durations. However, multimodal animal behavior classification can
help tackle this problem to some extent through using data of multiple sensing
modes, which may contain rich information about some rare behavior classes.

The alia class encompasses all behaviors apart from grazing, walking, resting,
and drinking. It includes many behaviors such as grooming, scratching,
suckling, calling (vocalizing), defecating, and urinating, which involve
various animal body poses and movement intensity levels. This makes the alia
class hard to distinguish from the other better-specified behavior classes.
The scarcity of training data for this class further adds to the challenge. In
future work, we will explore solutions that may obviate the need for defining
any such class.

##  7 Conclusion

We studied the classification of animal behavior via the joint use of relevant
data available through multiple sensing modes, i.e., accelerometry and GNSS,
using two approaches. One approach is based on concatenating the features
associated with all sensing modes and the other is based on fusing the
posterior probabilities of behavior classes inferred from the features
associated with each sensing mode. Both approaches are computationally
efficient and readily implementable on the embedded systems of the smart
collar and ear tags, which we use for monitoring cattle and classifying their
behavior in situ and in real time. Our cross-validated performance evaluations
using two datasets collected from grazing cattle via our collar and ear tags
revealed that both approaches result in improved classification accuracy
compared with using the data of either sensing mode alone. However, the second
approach, i.e., posterior probability fusion, led to superior classification
accuracy and efficiency despite its reliance on the assumption that the
features extracted from the data of different sensing modes are conditionally
independent given the behavior class.

## Acknowledgment

This research was undertaken with strategic investment funding from the CSIRO
and NSW Department of Primary Industries. We would like to thank the technical
staff who were involved in the research at CSIRO FD McMaster Laboratory
Chiswick, i.e., Flavio Alvarenga, Alistair Donaldson, and Reg Woodgate of the
NSW Department of Primary Industries, and Jody McNally and Troy Kalinowski of
the CSIRO Agriculture and Food. We also recognize the contributions of the
CSIRO Data61 staff who have designed and built the hardware and software of
the devices used for data collection, specifically, Lachlan Currie, John
Scolaro, Jordan Yates, Leslie Overs, and Stephen Brosnan.

## References

  * Arablouei et al. (2021) Arablouei, R., Currie, L., Kusy, B., Ingham, A., Greenwood, P.L., Bishop-Hurley, G.J., 2021\.  In-situ classification of cattle behavior using accelerometry data.  Computers and Electronics in Agriculture 183, 106045.
  * Arablouei et al. (2022) Arablouei, R., Wang, L., Currie, L., Alvarenga, F.A.P., Bishop-Hurley, G.J., 2022.  Animal behavior classification via deep learning on embedded systems.  arXiv preprint arXiv:2111.12295 .
  * Bailey et al. (2018) Bailey, D.W., Trotter, M.G., Knight, C.W., Thomas, M.G., 2018\.  Use of gps tracking collars and accelerometers for rangeland livestock production research.  Translational Animal Science 2, 81–88.
  * Brandes et al. (2021) Brandes, S., Sicks, F., Berger, A., 2021.  Behaviour classification on giraffes (giraffa camelopardalis) using machine learning algorithms on triaxial acceleration data of two commonly used gps devices and its possible application for their management and conservation.  Sensors 21, 2229\.
  * Brennan et al. (2021) Brennan, J., Johnson, P., Olson, K., 2021.  Classifying season long livestock grazing behavior with the use of a low-cost gps and accelerometer.  Computers and Electronics in Agriculture 181, 105957.
  * Cabezas et al. (2022) Cabezas, J., Yubero, R., Visitación, B., Navarro-García, J., Algar, M.J., Cano, E.L., Ortega, F., 2022.  Analysis of accelerometer and gps data for cattle behaviour identification and anomalous events detection.  Entropy 24, 336\.
  * Fogarty et al. (2021) Fogarty, E.S., Swain, D.L., Cronin, G.M., Moraes, L.E., Bailey, D.W., Trotter, M., 2021\.  Developing a simulated online model that integrates gnss, accelerometer and weather data to detect parturition events in grazing sheep: a machine learning approach.  Animals 11, 303\.
  * Fogarty et al. (2020) Fogarty, E.S., Swain, D.L., Cronin, G.M., Moraes, L.E., Trotter, M., 2020.  Behaviour classification of extensively grazed sheep using machine learning.  Computers and Electronics in Agriculture 169, 105175.
  * Greenwood et al. (2014) Greenwood, P.L., Valencia, P., Overs, L., Paull, D.P., Purvis, I.W., 2014.  New ways of measuring intake, efficiency and behaviour of grazing livestock.  Animal Production Science 54, 1796–1804.
  * Kamminga et al. (2018) Kamminga, J.W., Le, D.V., Meijers, J.P., Bisby, H., Meratnia, N., Havinga, P.J.M., 2018\.  Robust sensor-orientation-independent feature selection for animal activity recognition on collar tags.  Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 15\.
  * Kittler et al. (1998) Kittler, J., Hatef, M., Duin, R.P.W., Matas, J., 1998\.  On combining classifiers.  IEEE Transactions on Pattern Analysis and Machine Intelligence 20, 226–239.
  * Liu and Nocedal (1989) Liu, D.C., Nocedal, J., 1989\.  On the limited memory method for large scale optimization.  Mathematical Programming B 45, 503–528.
  * van der Maaten and Hinton (2008) van der Maaten, L.J.P., Hinton, G.E., 2008\.  Visualizing high-dimensional data using t-sne.  Journal of Machine Learning Research 9, 2579–2605.
  * Matthews (1975) Matthews, B.W., 1975.  Comparison of the predicted and observed secondary structure of t4 phage lysozyme.  Biochimica et Biophysica Acta (BBA) - Protein Structure 405, 442–451.
  * Pavlovic et al. (2021) Pavlovic, D., Davison, C., Hamilton, A., Marko, O., Atkinson, R., Michie, C., Crnojević, V., Andonovic, I., Bellekens, X., Tachtatzis, C., 2021\.  Classification of cattle behaviours using neck-mounted accelerometer-equipped collars and convolutional neural networks.  Sensors 21, 4050\.
  * Platt (1999) Platt, J., 1999.  Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods.  Advances in Large Margin Classifiers 10, 61–74.
  * Riaboff et al. (2022) Riaboff, L., Shalloo, L., Smeaton, A.F., Couvreur, S., Madouasse, A., Keane, M.T., 2022\.  Predicting livestock behaviour using accelerometers: A systematic review of processing techniques for ruminant behaviour prediction from raw accelerometer data.  Computers and Electronics in Agriculture 192, 106610.
  * Roy et al. (2022) Roy, A., Bertrand, S.L., Fablet, R., 2022.  Deep inference of seabird dives from gps-only records: Performance and generalization properties.  PLoS Computational Biology 18, e1009890.
  * Suparwito et al. (2021) Suparwito, H., Thomas, D.T., Wong, K.W., Xie, H., Rai, S., 2021.  The use of animal sensor data for predicting sheep metabolisable energy intake using machine learning.  Information Processing in Agriculture 8, 494–504.
  * Wang et al. (2021) Wang, L., Arablouei, R., Alvarenga, F.A.P., Bishop-Hurley, G.J., 2021\.  Animal behavior classification via accelerometry data and recurrent neural networks.  arXiv preprint arXiv:2111.12843 .
  * de Weerd et al. (2015) de Weerd, N., van Langevelde, F., van Oeveren, H., Nolet, B.A., Kolzsch, A., Prins, H.H.T., de Boer, W.F., 2015.  Deriving animal behaviour from high-frequency gps: tracking cows in open and forested habitat.  PLOS ONE 10, e0129030.
  * Williams et al. (2016) Williams, M.L., Parthaláin, N.M., Brewer, P., James, W.P.J., Rose, M.T., 2016.  A novel behavioral model of the pasture-based dairy cow from gps data using data mining and machine learning techniques.  Journal of Dairy Science 99, 2063–2075.