# BonnBot-I: A Precise Weed Management and Crop Monitoring Platform

Alireza Ahmadi, Michael Halstead, and Chris McCool All authors are with the
University of Bonn, Bonn 53115 Germany. {alireza.ahmadi, michael.halstead,
<EMAIL_ADDRESS>
###### Abstract

Cultivation and weeding are two of the primary tasks performed by farmers
today. A recent challenge for weeding is the desire to reduce herbicide and
pesticide treatments while maintaining crop quality and quantity. In this
paper we introduce BonnBot-I a precise weed management platform which can also
performs field monitoring. Driven by crop monitoring approaches which can
accurately locate and classify plants (weed and crop) we further improve their
performance by fusing the platform available GNSS and wheel odometry. This
improves tracking accuracy of our crop monitoring approach from a normalized
average error of $8.3\%$ to $3.5\%$, evaluated on a new publicly available
corn dataset. We also present a novel arrangement of weeding tools mounted on
linear actuators evaluated in simulated environments. We replicate weed
distributions from a real field, using the results from our monitoring
approach, and show the validity of our work-space division techniques which
require significantly less movement (a $50\%$ reduction) to achieve similar
results. Overall, BonnBot-I is a significant step forward in precise weed
management with a novel method of selectively spraying and controlling weeds
in an arable field.

Keywords — Robotics and Automation in Agriculture and Forestry; Agricultural
Automation; Field Robotics.

##  I Introduction

The preference for consuming more natural and organic foods has rapidly
increased in recent years [1]. This has forced the agricultural industry to
use fewer agri-chemicals when dealing with weeds while maintaining the quality
and quantity of the crop. Weed intervention is an important aspect of arable
farming due to its competition with crops for nutrients in the soil [2] which
potentially reduces yield. Currently, to alleviate this impact, the majority
of farmers use uniform treatments for weed control for instance by treating
the entire field with herbicide irrespective of weed presence. This approach
has led to an increasing number of herbicide-resistant weed species [3] as
well as negatively impacting the environment by increasing soil erosion and
water contamination [4].

Robotic weed intervention has the potential to revolutionize weeding paradigms
through plant-level weed management. For instance, by only treating a plant if
it is present and using the most appropriate action for the particular plant
species [5]. To achieve plant and species specific treatments, robots are
driven by advanced perception systems that can also provide rich crop
monitoring information [6]. Yet, robotic design has primarily explored how to
design the platform to perform weeding operation.

Several robotic weed control platforms have been introduced offering active
and passive interventions in field. A variety of weeding implements have been
investigated including physical [7], chemical [8], electrocuting [9], laser-
based [10]. Given the variety of tools, it is clear that there is no one best
solution and robotic solutions should be able to cater to a variety of tools.
The multi-modal approach of Bawden et al. [5] provided a clear step in this
direction, however, a downside of their approach was the need to densely
replicate each tool as they were mounted statically. We propose to overcome
this limitation by considering replicated movable tools.

Figure 1: BonnBot-I Platform, a robotic platform capable of conducting field
monitoring and precision weed management in arable field.

In this paper we introduce BonnBot-I, a robot that performs both field
monitoring and precision weed management. It enhances the capabilities of a
recently published field monitoring technique [6] by using multiple
localization sensors (GPS and odometry). The plant counting performance is
improved, reducing the normalized absolute error by more than half from
$8.3\%$ down to $3.5\%$. In developing BonnBot-I we also propose a novel
arrangement of weeding tools to enable precise weed management by replicating
linear systems. An advantage of this approach is that we can deploy both fewer
and smaller (more precise) tools, rather than densely replicated tools [5],
while still allowing for good coverage. We demonstrate that it is feasible to
have a system with just 4 replicated tools of size $0.05m$ on linear systems
to cover a width of $1.3m$ which would normally require at least $26$ non-
overlapping tools. This leads to the following novel contributions:

  1. 1.

We introduce BonnBot-I a fully autonomous precision weeding platform fully
compatible with ROS.

  2. 2.

Propose a new concept for weeding tools that enables flexible high-precision
weed management.

  3. 3.

Improve crop monitoring performance by exploiting the extra sensors available
with BonnBot-I.

  4. 4.

Release a new dataset consisting of corn as the crop, CN20. This is a
challenging dataset for crop monitoring approaches as it is a grass crop.

  5. 5.

Introduce a framework for testing different weeding intervention strategies
using a simulation environment.

##  II Related Work

Robotic platform based weed management techniques have rapidly evolved in the
last decade with the aim to treat each weed as precisely as possible. To
achieve this, plant-level intervention needs to operate in different fields
with varying crops, weed species and weed distributions. To enable this,
“smart farming techniques” aim to incorporate automated navigation [11], crop
monitoring [6], and weeding [12]. One of the key elements to achieve precise
weeding is plant-level treatment, where the treatment of each plant is
dictated by its species, size and its impact upon not only the crop but also
the environment [13]. Clearly, these approaches rely on the underlying
perception or agricultural monitoring approaches which have gained significant
research attention in recent years, including in glasshouses [14, 15],
orchards [16], and fields (for weed intervention) [17].

To achieve precise species level intervention recent methods have relied on
deep learning due to its accuracy and generalizability. Tian et al. [16]
compared both Faster-RCNN [18] and Yolo [19] and found Yolo to be superior in
both performance and speed, while, [20] was able increase both the performance
and speed of Faster-RCNN. Expanding on [21], [6] showed the viability of crop
agnostic monitoring by exploiting an augmented Mask-RCNN framework. To improve
the accuracy of monitoring techniques in arable farmland [22], was able to
utilize the temporal information captured by the platform. To selectively weed
[12] built, based on the work in [23], a fully convolutional network that was
able to accurately classify weeds in the field.

In order to perform intervention platforms are required, in recent years the
number of available robotic platforms for weed management has increased. An
early, low cost, platform was developed by [24] which contained a mechanical
weeding tool for intra-row intervention that required a human in the loop.
Bawden  _et al._ [5] proposed an automated platform that utilised a row of
weeding hoes and spray nozzles to improve on broadcast applications and
perform multi-modal weeding (physical and chemical). Also combining mechanical
tools (two ranks of stampers) and sprayers [12] is able to selectively weed
based on the overall size of the plant. To design a spraying platform [25]
developed a technique that was able to specifically target regions or weeds.
In an indoor experiment they were able to reduce the amount of chemicals used,
compared to a uniform sprayer, by $46.8\%$. Chang  _et al._ [26] built a
platform to evaluate two different mechanical weeding tools which was tested
on a purpose built field with a single crop-row of 20m. Their approach used
Yolo to both locate weeds of interest in the field but also control the
movement of the platform.

A consistent trend with the above approaches is that they aim to improve on
broadcast or uniform weeding by introducing equally spaced tools. In doing so
they ensure coverage but have the significant downside of having to replicate
the tool across the entire width of the robot. Yet, the optimal tool is
dependent upon the field and plants as shown by the variety of tools: physical
[7], chemical [8], electrocuting [9] or laser-based [10] implements. As such,
we conclude that to provide truly precise intervention a weeding platform
needs the ability to mount multiple tools. This issue exacerbates the problem
of having replicating equally spaced tools as two or more sets of tools have
to be replicated. An alternative is to have movable tools on a robot, but this
is only possible with an appropriate planning algorithm.

A frequently overlooked aspect for weeding is planning field based
intervention. If a robot carries multiple movable tools then planning their
deployment is essential, yet limited work has explored this aspect. Lee  _et
al._ [27] is one of the few works in this area. They presented a multi-query
approach for efficiently planning paths using a single UR5 robot manipulator
to enable precision weeding. This was achieved by maintaining a database of
pre-computed paths that was constructed offline and the optimal path was
chosen and adapted online. Xion  _et al._ [10] proposed a laser-based
intervention scheme that segmented the visible weeds into equally spaced
regions along the x-axis, this then allowed them to engage targets
sequentially.

In this work, we aim to alleviate some of the limitations of the previous
approaches by introducing a set of movable and replicated tools on a single
weeding platform. Furthermore, we present a robot that is able to navigate
down European standard crop planting patterns which automatically monitors the
crop with species level classification and localization. This provides both
automated field monitoring and weed management capabilities.

Figure 2: Overview of BonnBot-I with dimensions of key aspects such as height,
clearance, camera and tool placement (all numbers in centimeters).

##  III BonnBot-I

BonnBot-I is a platform capable of both field monitoring and weed management
developed at the University of Bonn. The base platform is a Thorvald system
[28] which is a lightweight four-wheel-drive (4WD) and four-wheel-steering
(4WS) system. With considerable modifications, we adapted this platform to an
arable farming and phenotyping robot suitable for operation in the real
phenotyping fields.

These modifications were carried out to ensure they met European farming
standards for the distance between rows. Based on these specifications the
width from wheel-centre-to-wheel-centre was set to $1.5m$. To ensure we could
monitor the different growth cycles of our primary crops (sugar beet, wheat,
and corn) the vertical clearance is set to $0.57m$. The length of BonnBot-I,
$1.39m$, was selected to ensure there was adequate space to install
replicating weeding tools. These dimensions also act to increase the stability
of the platform on the uneven terrain witnessed in arable farmland. An
overview of the BonnBot-I platform is provided in Fig. 2.

###  III-A Sensor Configuration

We equipped BonnBot-I with a range of sensors to perform both in-field weed
management and crop monitoring. There are two sets of sensors, the first set
is for “localization and navigation” and the second set is for “robotic
vision”.

####  III-A1 Localization

The localization of BonnBot-I is performed with a compact Inertial Navigation
System (INS), Ellipse2-D SBG Systems [29] which includes an IMU and a dual-
antenna receiver, multi-band GNSS receivers fixed at the front and back of the
robot at height of $1.85m$ above the ground. Using an on-board high-frequency
extended-Kalman filter fusion of IMU and GPS data provides us with a
horizontal and vertical position accuracy of $4cm$ and $3cm$ respectively.
Furthermore, the heading of the platform can be determined with an accuracy of
$0.1^{\circ}$ and $0.3^{\circ}$ in roll-pitch and yaw directions respectively.

####  III-A2 Robotic Vision

BonnBot-I is equipped a nadir-view cameras (Intel Real-sense D455) on the
front of the platform used for monitoring purposes. The Real-sense D455 is a
global shutter camera which provides RGB and registered depth images at
$15Hz$. On BonnBot-I it is fixed at a height of $0.78m$ providing a view-able
area of $1.4m\times 0.78m$ covering the gap between the two front wheels. This
is the sensor used to perform field monitoring.

###  III-B Weeding Implements

Achieving flexible, and repeatable weeding implements that can deploy a
variety of end-effectors is a key objective for BonnBot-I. This enables the
system to change tools given the current soil and weed populations. The
proposed design utilizes independently controlled high-resolution Igus linear
actuators fixed at height of $0.72m$ above the ground and creating a working
space of $1.3m\times 0.36m$; the current design uses 4 such linear actuators.
Each linear axis has a length of $1.3m$ and is controlled by an Igus dryve D1
motor control system via Modbus connection with a maximum resolution of
$0.01cm$ and is capable of performing translations with maximum velocity and
acceleration of $5m/s$ and $10m/s^{2}$, respectively. All linear axes are
equally-spaced and currently carry spot-spray nozzles, however, the system
design permits many kinds of end-effectors, such as mechanical hoeing,
providing flexibility.

To control the linear actuators and spray valves we use a ROS operated
Raspberry-Pi 3B and to ensure minimal action delay the nozzles are accessed
via high speed N-channel MOSFET-transistors. Hence, ultimate operation time
for each spray head in our system adds up to $10\sim 12$ms including valves
On-Off time. The spray system consists of a reservoir tank capable of carrying
$5L$s of compressed liquid with a maximum $16bar$ pressure, as well as a
compact $8bar$ portable compress which is fixed on the robot. As the droplet
size from the spray nozzles depends on the liquid pressure we use individually
adjustable valves. This allows us to control the spray footprints on the
ground individually for each nozzle between $0.02m$ to $0.13m$; for this paper
we assume a constant spray footprint of $0.05m$.

###  III-C In-Field Intervention Pipeline

All sensors on BonnBot-I are directly connected to a high performance fanless
embedded computer (DS-1202) powered by 7th Generation Intel® Core™ processor
running the robot operating system (ROS). This computer is equipped with an
Nvidia® Quadro P2200 featuring a Pascal GPU with 1280 CUDA cores, 5 GB GDDR5X
on-board memory which is used for parallel processing and inference of DNNs.
Additionally, a dedicated Intel® NUC PC runs the controller for the platforms
motion providing wheel odometry data and the status of batteries and
electronic infrastructure on the platform.

The software architecture of BonnBot-I contains four different nodes enable
selective in-motion intervention. (I) Field Monitoring: runs Mask-RCNN for
instance-based semantic segmentation and intra-camera tracking which estimates
necessary phenotypic information about the plants, more details are in Sec.
IV. (II) In-field localization: improves the localization accuracy of the
robot using an EKF to fuse GPS and wheel odometry data [30]. (III)
Intervention Controller: manages the targets within the work-space of the
weeding tools and includes planning paths for the intervention heads (sprays)
described in Sec. V. (IV) Weeding Implement: provides low-level control of the
weeding tool (e.g. actuation) by taking commands from the intervention
controller.

###  III-D Weeding Simulation Framework

Figure 3: BonnBot-I ROS simulation with active sensors and actuators (left),
An example view of weeding simulation Framework (right) with green markers as
plants and red markers representing weeds in each segment. A weeding path is
also visualized in blue line for single axis weeding scenario.

Conducting experiments in real fields is potentially time consuming and
costly. An accurate and reliable simulation environment that mimics real world
situations is an invaluable tool which avoids this issue. We use two different
types of simulation environments for development and evaluation of our methods
on BonnBot-I a ROS based one-to-one scale simulation and a native python
simulator especially developed for weeding application and intervention
motoring purposes. A demonstration of the ROS-based simulation model is shown
in Fig. 3-(left), where all sensors and actuators are active. Then a field
generator allows us to create row-crop fields with different plant sizes and
crop-row shapes, simulating a real field. To evaluate the weeding algorithms,
we developed a native Python based framework capable which simulates the robot
kinematics and generates synthetic crop-rows with varying weed distributions.
This framework uses Open3D and Pyglet python libraries for rendering graphics,
an simplified example view of the planning scenario is shown in Fig.
3-(right). We used this environment to implement and evaluate different
weeding strategies which we elaborate on in Sec. V.

##  IV Field Monitoring

One of the key benefits of BonnBot-I as a platform is that it can monitor the
state of the field while traversing it, this supplements its key function as a
weeding platform. We demonstrate the potential of BonnBot-I for field
monitoring by illustrating how the extra localization sensors can enhance the
existing tracking algorithms in our prior work [6]. This approach used Mask-
RCNN to provide instance-based segmentation, species-level information (e.g.
crop and weed species) and the viewable surface area. This approach included a
tracking-via-segmentation technique that outlined the benefit of of spatial
matching operator, coined dynamic radius (DR), over a pixel-wise version,
referred to as intersection over union (IoU). This technique also exploited
re-projection between frames using wheel odometry and camera parameters. This
enabled more accurate tracking of objects in the scene, however, wheel
odometry is often prone to errors. Using the extra GPS sensors available on
BonnBot-I has the potential to increase performance by re-projecting more
accurately between frames.

In this paper we demonstrate that the extra localization sensors on BonnBot-I
can be used to enhance the performance of field monitoring. In particular, we
fuse the available odometery and GPS information with an EKF [30]. This
algorithm recursively estimates the state of a non-linear system in an optimal
way. Furthermore, adding local source of motion estimation can considerably
reduce the risk of outage due to lack of proper satellite observations [11].

In our prior work [6] we concentrated on sweet pepper in a horticultural
setting and sugar beet in arable farmland. In both cases the objects we aim to
detect are somewhat robust to external influences such as weather conditions.
However, some of the weed species witnessed in the arable farmland were
grasses and their accurate localization proved difficult. In this paper we
further outline the ability for our approach to be crop agnostic by performing
monitoring on a novel grass crop dataset consisting of corn. Corn has a long
leaf structure which makes it susceptible to weather conditions resulting in a
difficult crop to localize, due to this, in Sec. VI-A we outline our
performance using both the pixel-wise and spatial matching criteria from [6].

This corn data set (CN20) was acquired using BonnBot-I from a phenotyping
field at campus Klein-Altendorf (CKA) of the University of Bonn. The data was
captured using an Intel RealSense D435i sensor with a nadir view of the ground
in front of the robot and resolution of $1280\times 720$ with a frame-rate of
$15Hz$. The non-overlapping training, validation, and evaluation data includes
RGB-D frames for $170$, $43$ and $70$ images respectively. This data comes
from six different rows providing unique crop and weed distributions due to
the non-homogeneous growth stage of the weeds. In total there are nine
different categories of weeds containing a total of $2566$ and $1261$
instances of crop and weeds respectively. The data is annotated to include
instance based pixel-wise segmentation, bounding boxes and stem locations of
each instance in Coco format [31]. Fig. 4 shows an example annotated image of
CN20 dataset.

---
Figure 4: Example image of the CN20 dataset. Left is the original images and
right is the same scene with instance-based annotations with unique color for
each plant.

##  V Selective Precise In-Field Intervention

BonnBot-I is equipped with a novel weeding tool design enabling high precision
plant-level field interventions. It consists of a set of replicated linear
actuators and is controlled via the intervention controller unit consisting of
several components which are elaborated in the following sections. We briefly
explain the conceptual design of the weeding tool, its requirements and
operation assumptions. Then, we introduce our method for managing targets in
the work-space of the weeding tools. Finally, we elaborate on path planning
strategies used for controlling intervention heads in action.

###  V-A Plant-level Treatment In Field

We assume the robot moves along a crop-row with constant speed $\gamma$.
Consequently, intervention is time-critical and must respect the spatial
ordering of the weeds. There is a constant gap ($\Gamma$) between the tools
and the area sensed by the camera ($C_{detect}$). Similar to [32] we assume
weeds are uniformly distributed in the field with density $\lambda$
weeds/$m^{2}$. Hence, using a Poisson process we can explain the distance
between the weeding implement and individual weeds by accounting for the
arrival rate of $\eta=\lambda\times\Pi$. We use the motion along the $x$-axis
of the robot frame $\mathcal{F}_{R}$ to explain the weeds interval distance
($\delta_{x}$), visualized in Fig. 5. This can be shown using the following
probability density function,

$f(\delta_{x})=\lambda\Pi e^{-\lambda\Pi\delta_{x}},$ (1)

also the location of weeds on the $y$ axis can be represented via a uniformly
distributed random variable $y$ as,

$f(y)=\left\\{\begin{array}[]{cc}\frac{1}{\Pi}&for\ \ 0\leq y\leq\Pi\\\
0&otherwise\end{array}\right..$ (2)

To engage the $i$-th intervention head with the $j$-th weed it has to
traverse,

$\delta^{ij}_{y}=|\textit{h}_{i}-\textit{n}_{j}|,\text{ where
}0\leq\delta^{ij}_{y}\leq\Pi,$ (3)

where $\textit{h}_{i}$ is the current position of $i$-th intervention head and
the $\textit{n}_{j}$ denotes to the position of the $j$-th weed. Therefore,
the probability of visiting the $j$-th weed with $i$-th intervention head can
be calculated with,

$\textit{P}_{ij}=\textit{P}\left(\dfrac{\gamma}{\vartheta}<\dfrac{\delta_{x}}{\delta_{y}}\right),$
(4)

where $\vartheta$ denotes to the maximum velocity of linear axes. We assume
all targets are detected by the time they reach the bottom edge of the
camera’s viewable area.

Figure 5: Visualization of Kinematic model of weeding tool work-space (a) the
weeds detected in the viewable area of the camera $C_{detect}$ (c); and (b)
the gap between two regions.

###  V-B Target-Space Management

Planning the motion of each intervention head must be done prior to targets
entering the weeding tool work-space. In the proposed workflow, the
intervention controller node receives the detected targets at time
$t+\tau_{d}$ where $\tau_{d}$ is the time required for detection in the
monitoring node. The monitoring node provides plant specific information like:
plant category, pixel-wise segmentation, estimated area, and the bounding box.
Furthermore, we estimate plant centers based on provided bounding box in the
scene. This information is then used in the target-space management step to
assign targets to the intervention heads. Based on this the next step finds
the best motion plan for each intervention head by maximizing the number of
targets that are visited (sprayed).

Let $\mathcal{H}$ denote the number of independent controllable intervention
heads and $\mathcal{N}$ be the number of targets that appear under the robot.
We use a uni-directional constrained node-graph to model the targets-space.

To obtain the global spatial order of targets in a segment we use the
$\delta_{x}$ of each weed (see Fig. 5). In Fig. 6(a), each node (circle) shows
a weed along with the connecting path between nodes $j$ to $k$ represented
with a uni-directional link (arrow) $l_{jk}$. The link $l_{jk}$ exists if,
node $j$ geometrically is located after node $k$ in the 3D world frame
$\mathcal{F}_{w}$ in the direction of motion. Furthermore, the link $l_{jk}$
is associated with an inter-weed cost $\varrho_{jk}$ based on the distance of
nodes $j$ and $k$ and a property denoting motion probability of
$\textit{P}_{jk}$ based on Eq. (4). We calculate inter-weed costs using the
top-right of the cost-matrix $\mathcal{G}_{\mathcal{N}\times\mathcal{N}}$ (to
respect the weeds spatial order).

There are $\mathcal{H}$ independent interventions heads and so multiple plans
which can lead to the same number of targets being visited (sprayed). To solve
this problem, we consider the weeds as a sets of targets detected in one
location, this motivates us to assign intervention targets to the
$\mathcal{H}$ heads as either distance-based or work-space division-based
assignments.

  1. 1.

Distance-based Target Assignment (D): In this approach, target $j$ gets
assigned to the laterally closest intervention head along the sliding
direction ($y$-axis). This means, selected intervention head $i$ has the least
motion required to reach the weed $j$. The lateral distance between heads and
weeds are defined based on 2D euclidean distance between projection of
intervention head’s position on ground plane and weed positions on same plane
w.r.t the $\mathcal{F}_{w}$ frame.

  2. 2.

Static Work-space Division-based Target Assignment (SD): In this method, we
divide the work-space of weeding tool to $\mathcal{H}$ sub-sections of width
$\Pi/\mathcal{H}$ meters. Hence, each intervention head is only responsible
for engaging with weeds laying within it’s sub-work-space as shown in Fig.
6(b)-top.

  3. 3.

Dynamic Work-space Division-based Target Assignment (DD): In this model, for
each new set of detected weeds we first determine the minimum region of
intervention defined by $y_{min}$ and $y_{max}$ (see Fig. 6 (b)). The minimum
region of intervention is then divided into $\mathcal{H}$ equal sub-regions.
This process assists in optimizing the planning for weed engagement by
potentially reducing the area any one tool has to cover.

Figure 6: In (a) an example unidirectional constrained node-graph is presented
where the numbered circles represent weeds (targets) for one intervention tool
$n_{1}$ with different possible start paths (in different colors). In (b) we
present a visualization of static work-space sub-division (top) and dynamic
work-space sub-divisions (bottom).

###  V-C Intervention Heads Route Planning

Here we address how to plan $\mathcal{H}$ independent efficient routes. The
planned routes must guide intervention heads through all their assigned
targets while minimizing the chance of missing any target. This has to take
into account the the prior knowledge of an intervention head’s position, robot
linear speed as well as the limits of speed and acceleration of linear axes.

The planning approach generates $m$ potential trajectories
$\vec{\mathbf{T}}=[\vec{T}_{0},\dots,\vec{T}_{m}]$ for each intervention head.
Each trajectory $\vec{T}_{i}$ is an ordered list of length $q$ consisting of
weed positions which can be visited. To obtain $\vec{\mathbf{T}}$ we use the
following approaches:

  1. 1.

Brute-Force: In this case we compute all possible routes by finding the
permutation of all nodes in the graph (without considering the direction of
links). Then the routes with the lowest cost and maximum success rate will be
selected from all predicted routes.

  2. 2.

Open Loop Traveling Salesman Planning: This approach, termed OTSP, is a
variant of the classic travelling salesman problem where the agent must visit
all nodes of a graph once without making a loop back (Hamilton loop) to the
start node [33]. To solve this we use an approach similar to $n$OTSP where the
agent only needs to visit $n$ nodes in the graph, however, in our problem
setting we aim to maximize the number of visited nodes while considering other
important criteria like cost and success rate. We use our constrained uni-
directional node-graph representation as a base for solving $n$OTSP using
dynamic programming.

The optimal trajectory for each intervention head is obtained by considering
two criteria: number of nodes successfully visited and the total movement of
the predicted trajectory. In every trajectory, we calculate the number of
nodes that satisfy Eq. (4) to determine if a node can be successfully visited.
This gives us an updated set $\vec{\mathbf{T}^{\prime}}$ which only consists
of nodes in the trajectories which are feasible. From this updated set
$\vec{\mathbf{T}^{\prime}}$ we then calculate the movement cost-matrix
$\mathcal{G}$,

$\mathcal{G}(\vec{T}^{\prime}_{i})=\sum_{j=0}^{q-1}(n_{j}-n_{j+1})^{2}.$ (5)

After this process, the trajectory from $\vec{\mathbf{T}^{\prime}}$ with the
maximum number of successfully vised nodes is passed to the intervention
controller. In the case multiple trajectories successfully visit the same
number of nodes, the trajectory which also minimizes the movement cost will be
passed to the intervention controller.

##  VI Experimental Evaluations

###  VI-A Field Monitoring

We evaluate the performance improvements to field monitoring techniques when
using a range of localization sensors. Our previous work [6] is enhanced by
fusing the wheel odometry and GNSS, which are available on the platform, using
EKF. With this information we are able to improve the average normalized
absolute error (NAE) across the three CN20 evaluation rows using our dynamic
radius (DR) tracker from $8.3\%$ to $3.5\%$, more than halving the NAE. This
is a considerable improvement as the absolute error of the field tracking
system is less than $5\%$ NAE. We attribute this performance improvement to
better localization information supplied through multiple localization sensors
and sensor fusion. To fully appreciate the impact of sensor fusion using EKF
we also explore the improvements for a pixel-wise matching tracker (IoU). This
is much simpler tracking approach, although susceptible to leaf movement due
to wind, is greatly improvement with the NAE improving from $71.7\%$ to
$25.7\%$ with the incorporation of multiple localization sensors and sensor
fusion. Finally, in our experiments we were consistently able to achieve
$R^{2}$ scores greater than $0.95$, showing that our tracker was linearly
correlated with the ground truth. These results demonstrate the importance of
multiple localization sensors and the considerable positive impact they can
have on existing field monitoring techniques.

###  VI-B Weeding Planning Performance

To evaluate the performance of the weeding tool planning explained in Sec. V,
we use our native python simulator (Sec. III-D) specifically designed for this
purpose. We investigate the performance of our approach using two different
types of crop-row models: simulated rows of crops/weeds and real plant
distributions predicted from our monitoring systems for both SB20 and CN20
datasets evaluation rows. We initially evaluated the performance of the Brute-
Force planner against the $n$OTSP planner and found the results where similar
in trivial cases with a limited number of targets ($\mathcal{N}\leq 10$).

The computational expense of the Brute-Force method increases almost
exponentially when there are more than four targets as it has a run-time
complexity of $O(n!)$ in comparison with $O(n^{2}2^{n})$ of graph based
$n$OTSP. This is a significant problem for weeding applications where real-
time performance is important. In the case where we see only ten weeds in the
planning region the Brute-Force approach requires $3.7$s compared to $n$OTSP
which requires only $266\mu$s. This is a prohibitive quality of the Brute-
Force approach and for our two weeding experiments we employ the $n$OTSP
technique.

####  VI-B1 Planning on Simulated Crop-Rows

In our first experiment we evaluate the performance of different weed
densities and a different number of linear axes. This evaluation uses the
simulation environment introduced in Sec. III-D, allowing us to control the
experimental parameters. For our platform hyper-parameters we keep a constant
robot speed of $\gamma=0.5$ and set the velocity of the linear actuators to
$\vartheta=5$. The field parameters are set to $3$ crops-rows in a single lane
with a length of $20$m. To fully analyse the performance of our approach we
vary the weed density such that $\lambda\,=\left[3,5,10,20,40\right]$
representing the weeds per $m^{2}$. Finally, to outline the benefit of having
multiple linear axes we show the performance for
$\mathcal{H}\,=\left[1,2,4,8\right]$.

The results for this simulated experiment are summarized in Fig. 7, where we
provide the comparison of weed density to the percentage of missed targets.
From this figure it is evident that increasing the number of linear axes has
an obvious impact on results, with $8$ axes performing better than all others.
We see from this that even with a distribution of $40$ weeds (the hardest
case) the worst performing $8$ axes system, Distance-Based, achieves a loss of
$\sim{15\%}$.

Overall, the Distance-Based approach routinely performs worse as the weed
density increases, this is particularly evident as the number of heads is
increased. This will be further evaluated in the next evaluation, but as the
distribution grows the intervention heads need to travel further to meet the
demands of the planner, this movement can negatively impact the capacity for
the intervention head to reach the next weed. Finally, the two division based
methods appear to perform at a commensurate level across all weed densities in
our simulations. We attribute this to the wide distribution of weeds negating
the impact of the dynamic approach, meaning, weeds generally appear across the
entire lane rather than concentrated in a specific region. Overall, this shows
the validity of our planning methods for weed intervention in a uniformly
distributed pattern.

Figure 7:  The results of the $n$OTSP weeding planner. Loss represents the
number of missed weeds. We evaluate for 5 different weed density models (x
axis); three different planning types: Dist.-Based (D), Dyn.-Div. (DD), and
Sub-Div. (SD); and intervention heads $\mathcal{H}=1,2,4,8$, yellow, green,
purple and blue, respectively.

####  VI-B2 Planning on Real Crop-Rows

To further evaluate the performance of our three planning approaches, we use
real weed distributions captured in the fields within weeding simulator. This
information is obtained from the crop monitoring approach outlined in Sec. IV
and aggregated into a simulated row. We perform this on the evaluation rows
for both the SB20 and CN20 datasets where we have three different weed
distributions: low (SB20-S2), medium (CN20), and high (SB20-S1).

The results of our three planning approaches are displayed in Tab. I, in this
experiment we only use four intervention heads as this accurately evaluates
BonnBot-I performance. Tab. I displays two metrics, first, the percentage of
missed weeds, and second, the mean and standard deviation of the distance
moved by the axes.

TABLE I: The rate of Loss ($\%$) and average traveled distance ($m$) of
interventions heads in real-world weeding scenarios.

| Dist.-Based (D) |  Sub-Div. (SD) | Dyn.-Div. (DD)
---|---|---|---
| (%) | (m) | (%) | (m) | (%) | (m)
CN20 | 0.0 | 2.7$\pm$0.2 | 4.3 | 2.7$\pm$2.8 | 3.4 | 4.0$\pm$3.9
SB20-S1 | 11.9 | 10.1$\pm$0.9 | 19.8 | 4.7$\pm$3.5 | 13.5 | 5.0$\pm$1.8
SB20-S2 | 0.0 | 1.4$\pm$0.2 | 2.3 | 1.5$\pm$0.6 | 0.0 | 1.0$\pm$0.8

For the percentage of missed weeds we see that the high (SB20-S1) level of
weeds is the most difficult to intervene on, this is somewhat expected due to
the heavy distribution of weeds. However, the simple Distance-Based approach
outperforms the other two approaches for this distribution which is similar to
that seen in Fig. 7 for a density of $10$ per $m^{2}$. The poorer performance
of the two division based approaches can be attributed to the distribution of
the weeds in a real crop-row. While, we performed the Chi-squared test on our
rows and found no evidence to suggest they were not uniform we still assert
that some sections of the row do not conform to the uniform property,
increasing complexity. This performance is mirrored through the other rows,
where the Distance-Based approach achieves higher scores. However, once we
reduce the weed density (SB20-S2, CN20) we are able to achieve a percentage of
missed weeds close to zero for all approaches.

Our final evaluation is based on the movement requirements of the different
planning approaches, where a value closer to zero is desired. While, Distance-
Based achieved the best performance for the percentage of missed weeds we see
the negative aspect of this approach here. Both the division based approaches
are able achieve considerably better results, in fact for SB20-S1 both
division based approaches move half as much as the Distance-Based. Even with
considerably more standard deviation these two approaches outline the benefit
of sub-sampling the work-space for more efficient tool usage. We believe that
in the future this work can be used to provide baseline information about
robotic path planning for weeding applications. This includes potential
improvements by jointly minimizing the total distance travelled (of the
intervention heads) and the loss.

##  VII Conclusion

In this paper, we introduced BonnBot-I a crop monitoring and weeding platform
specifically designed for European phenotyping fields. We showed how BonnBot-I
can greatly improving crop monitoring, reducing the NAE from $8.3\%$ to
$3.5\%$, by combining the extra localization sensors it has available.
Furthermore, we present a novel arrangement of weeding tools mounted on linear
actuators along with three associated target planning approaches. These target
planning approaches were evaluated in simulated environments which are able to
mirror real-world weed distributions by using the results from our field
monitoring approach. Experiments on these simulated real-world fields
demonstrated the validity of our proposed work-space division techniques and
showed that they led to significantly less movement ($10m$ compared to $5m$)
when compared to distance-based target assignment. Finally, we found that
while the real-world data was generally uniform in future any planning systems
should be evaluated on real-world weed distributions.

## Acknowledgements

This work was funded by the Deutsche Forschungsgemeinschaft (DFG, German
Research Foundation) under Germany’s Excellence Strategy - EXC 2070 –
390732324.

## References

  * [1] J. Blasco, N. Aleixos, J. Roger, G. Rabatel, and E. Moltó, “Ae—automation and emerging technologies: Robotic weed control using machine vision,” _Biosystems Engineering_ , vol. 83, no. 2, pp. 149–157, 2002.
  * [2] D. Slaughter, D. Giles, and D. Downey, “Autonomous robotic weed control systems: A review,” _Computers and electronics in agriculture_ , vol. 61, no. 1, pp. 63–78, 2008.
  * [3] . . A. w. I. Heap. Online. Internet. Friday, February 25, “The international survey of herbicide resistant weeds,” Online: www.weedscience.com, 2022\.
  * [4] M. J. Mia, F. Massetani, G. Murri, and D. Neri, “Sustainable alternatives to chemicals for weed control in the orchard–a review,” _Horticultural Science_ , vol. 47, no. 1, pp. 1–12, 2020.
  * [5] O. Bawden, J. Kulk, R. Russell, C. McCool, A. English, F. Dayoub, C. Lehnert, and T. Perez, “Robot for weed species plant‐specific management,” _Journal of Field Robotics_ , vol. 34, pp. 1179–1199, 2017.
  * [6] M. Halstead, A. Ahmadi, C. Smitt, O. Schmittmann, and C. Mccool, “Crop agnostic monitoring driven by deep learning,” _Frontiers in Plant Science_ , p. 2937, 2021.
  * [7] C.-L. Chang, B.-X. Xie, and S.-C. Chung, “Mechanical control with a deep learning method for precise weeding on a farm,” _Agriculture_ , vol. 11, no. 11, p. 1049, 2021.
  * [8] X. Wu, S. Aravecchia, P. Lottes, C. Stachniss, and C. Pradalier, “Robotic weed control using automated weed and crop classification,” _Journal of Field Robotics_ , vol. 37, no. 2, pp. 322–340, 2020.
  * [9] J. Ascard, P. Hatcher, B. Melander, M. Upadhyaya, and R. Blackshaw, “10 thermal weed control,” _Non-chemical weed management: principles, concepts and technology_ , pp. 155–175, 2007.
  * [10] Y. Xiong, Y. Ge, Y. Liang, and S. Blackmore, “Development of a prototype robot and fast path-planning algorithm for static laser weeding,” _Computers and Electronics in Agriculture_ , vol. 142, pp. 494–503, 2017.
  * [11] A. Ahmadi, M. Halstead, and C. McCool, “Towards autonomous crop-agnostic visual navigation in arable fields,” _arXiv preprint arXiv:2109.11936_ , 2021\.
  * [12] A. Pretto, S. Aravecchia, W. Burgard, N. Chebrolu, C. Dornhege, T. Falck, F. V. Fleckenstein, A. Fontenla, M. Imperoli, R. Khanna _et al._ , “Building an aerial–ground robotics system for precision farming: An adaptable solution,” _IEEE Robotics & Automation Magazine_, vol. 28, no. 3, pp. 29–49, 2020.
  * [13] C. Blaix, A. C. Moonen, D. Dostatny, J. Izquierdo, J. Le Corff, J. Morrison, C. Von Redwitz, M. Schumacher, and P. Westerman, “Quantification of regulating ecosystem services provided by weeds in annual cropping systems using a systematic map approach,” _Weed research_ , vol. 58, no. 3, pp. 151–164, 2018.
  * [14] C. Smitt, M. Halstead, T. Zaenker, M. Bennewitz, and C. McCool, “Pathobot: A robot for glasshouse crop phenotyping and intervention,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2021, pp. 2324–2330.
  * [15] C. Smitt, M. Halstead, A. Ahmadi, and C. McCool, “Explicitly incorporating spatial information to recurrent networks for agriculture,” _IEEE Robotics and Automation Letters_ , 2022.
  * [16] Y. Tian, G. Yang, Z. Wang, H. Wang, E. Li, and Z. Liang, “Apple detection during different growth stages in orchards using the improved yolo-v3 model,” _Computers and electronics in agriculture_ , vol. 157, pp. 417–426, 2019.
  * [17] A. Peruzzi, L. Martelloni, C. Frasconi, M. Fontanelli, M. Pirchio, and M. Raffaelli, “Machines for non-chemical intra-row weed control in narrow and wide-row crops: a review,” _Journal of Agricultural Engineering_ , vol. 48, no. 2, pp. 57–70, 2017.
  * [18] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” in _Advances in neural information processing systems_ , 2015, pp. 91–99.
  * [19] J. Redmon and A. Farhadi, “Yolov3: An incremental improvement,” _arXiv preprint arXiv:1804.02767_ , 2018.
  * [20] S. Wan and S. Goudos, “Faster r-cnn for multi-class fruit detection using a robotic vision system,” _Computer Networks_ , vol. 168, p. 107036, 2020.
  * [21] M. Halstead, S. Denman, C. Fookes, and C. McCool, “Fruit detection in the wild: The impact of varying conditions and cultivar,” in _Proceedings of Digital Image Computing: Techniques and Applications (DICTA)_ , 2020.
  * [22] A. Ahmadi, M. Halstead, and C. McCool, “Virtual temporal samples for recurrent neural networks: applied to semantic segmentation in agriculture,” in _German conference on pattern recognition_ , 2021.
  * [23] P. Lottes, J. Behley, A. Milioto, and C. Stachniss, “Fully convolutional networks with sequential information for robust crop and weed detection in precision farming,” _IEEE Robotics and Automation Letters_ , vol. 3, no. 4, pp. 2870–2877, 2018.
  * [24] M. Pérez-Ruíz, D. C. Slaughter, F. A. Fathallah, C. J. Gliever, and B. J. Miller, “Co-robotic intra-row weed control system,” _Biosystems Engineering_ , vol. 126, pp. 45–55, 2014. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1537511014001214
  * [25] M. Zhou, H. Jiang, Z. Bing, H. Su, and A. Knoll, “Design and evaluation of the target spray platform,” _International Journal of Advanced Robotic Systems_ , vol. 18, no. 2, p. 1729881421996146, 2021.
  * [26] C.-L. Chang, B.-X. Xie, and S.-C. Chung, “Mechanical control with a deep learning method for precise weeding on a farm,” _Agriculture_ , vol. 11, no. 11, 2021. [Online]. Available: https://www.mdpi.com/2077-0472/11/11/1049
  * [27] J. J. H. Lee, K. Frey, R. Fitch, and S. Sukkarieh, “Fast path planning for precision weeding,” in _Australasian Conference on Robotics and Automation, ACRA_ , 2014.
  * [28] L. Grimstad and P. J. From, “Thorvald ii-a modular and re-configurable agricultural robot,” _IFAC-PapersOnLine_ , vol. 50, no. 1, pp. 4588–4593, 2017.
  * [29] SBG Systems, “Technical Data Ellipse2-D,” 2020, https://www.sbg-systems.com/products/ellipse-series/#ellipse-d_rtk_gnss_ins.
  * [30] L. Wei, C. Cappelle, Y. Ruichek, and F. Zann, “Intelligent vehicle localization in urban environments using ekf-based visual odometry and gps fusion,” _IFAC Proceedings Volumes_ , vol. 44, no. 1, pp. 13 776–13 781, 2011.
  * [31] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _European conference on computer vision_. Springer, 2014, pp. 740–755.
  * [32] O. Bawden, J. Kulk, R. Russell, C. McCool, A. English, F. Dayoub, C. Lehnert, and T. Perez, “Robot for weed species plant-specific management,” _Journal of Field Robotics_ , vol. 34, no. 6, pp. 1179–1199, 2017.
  * [33] H. H. Chieng and N. Wahid, “A performance comparison of genetic algorithm’s mutation operators in n-cities open loop travelling salesman problem,” in _Recent Advances on Soft Computing and Data Mining_. Springer, 2014, pp. 89–97.


# Analyzing COVID-19 Vaccination Sentiments in Nigerian Cyberspace: Insights
from a Manually Annotated Twitter Dataset
††thanks:  Funding received from HausaNLP and Arewa Data Science Academy.

Ibrahim Said Ahmad  Northeastern University
Boston, USA
<EMAIL_ADDRESS>Lukman Jibril Aliyu  HausaNLP
Kano, Nigeria
<EMAIL_ADDRESS>Auwal Abubakar Khalid  Bayero University
Kano, Nigeria
<EMAIL_ADDRESS>Saminu Muhammad Aliyu  Bayero University
Kano, Nigeria
<EMAIL_ADDRESS>Shamsuddeen Hassan Muhammad  University of Porto
Porto, Portugal
<EMAIL_ADDRESS>Idris Abdulmumin  Ahmadu Bello University
Zaria, Nigeria
<EMAIL_ADDRESS>Bala Mairiga Abduljalil  University Of Maiduguri
Borno, Nigeria
<EMAIL_ADDRESS>Bello Shehu Bello  Bayero University
Kano, Nigeria
<EMAIL_ADDRESS>Amina Imam Abubakar  University of Abuja
Abuja, Nigeria
<EMAIL_ADDRESS>
###### Abstract

Numerous successes have been achieved in combating the COVID-19 pandemic,
initially using various precautionary measures like lockdowns, social
distancing, and the use of face masks. More recently, various vaccinations
have been developed to aid in the prevention or reduction of the severity of
the COVID-19 infection. Despite the effectiveness of the precautionary
measures and the vaccines, there are several controversies that are massively
shared on social media platforms like Twitter. In this paper, we explore the
use of state-of-the-art transformer-based language models to study people’s
acceptance of vaccines in Nigeria. We developed a novel dataset by crawling
multi-lingual tweets using relevant hashtags and keywords. Our analysis and
visualizations revealed that most tweets expressed neutral sentiments about
COVID-19 vaccines, with some individuals expressing positive views, and there
was no strong preference for specific vaccine types, although Moderna received
slightly more positive sentiment. We also found out that fine-tuning a pre-
trained LLM with an appropriate dataset can yield competitive results, even if
the LLM was not initially pre-trained on the specific language of that
dataset.

###### Index Terms:

Covid-19, Vaccination, Sentiment Analysis, Natural Language Processing, Large
Language Models

##  I Introduction

Over the last decade, social media has provided abundant data that researchers
often utilized to derive insights into people’s sentiments. As such, several
research areas have emerged relying on social media data, including sentiment
analysis, emotion detection, stance detection and social network analysis [1].
In this research, we explored the use of tweets in Nigerian cyberspace for the
prediction of people’s opinions towards the COVID-19 vaccine. This approach is
practical because it allows for real-time feedback, is cost-effective,
provides a large reach, offers unfiltered opinions, and enables trend
analysis.

The COVID-19 pandemic is a global health crisis caused by the spread of the
novel coronavirus SARS-CoV-2 [2]. It has had far-reaching and profound impacts
on societies, economies, healthcare systems, and daily life worldwide. The
pandemic had a significant impact on public health and society at large,
necessitating the rapid development and distribution of vaccines to combat the
virus. While the availability of vaccines is crucial, so too is public
sentiment towards them [3, 4, 5]. In Nigeria, social media platforms such as
Twitter have played a crucial role in shaping public discourse around COVID-19
[6, 7]. As such, the discussions on Twitter provide crucial data that can be
used to study COVID-19 vaccination in Nigeria. Therefore, we crawl tweets in
the Nigerian cyberspace using the discontinued Twitter Academic
API111https://www.cnn.com/2023/04/05/tech/academic-researchers-blast-Twitter-
paywall/index.html from 1st November 2020 to 30th June 2022 using COVID-19
vaccine keywords. A total of 5200 tweets were collected. We preprocessed the
tweets by removing URLs, tweets with less than three words, and tweets not in
Latin characters, so the tweets were reduced to 4320. Therefore 4,320 tweets
were manually annotated by three (3) independent annotators.

We conducted exploratory data analysis to derive insights from the dataset.
The results of the analysis reveal that the majority of Nigerians either
believe that vaccines are secure and life-saving or hold a neutral stance on
the vaccines, while a smaller fraction of Nigerians have expressed concerns
about the vaccine’s safety. Additionally, we found that the majority of the
people in Nigeria hold a neutral stance on COVID-19 brands, notwithstanding, a
small percentage of the people still hold a negative stance towards the
brands.

We conducted experiments to demonstrate that a strong performance can be
achieved on COVID-19 acceptance prediction using the developed dataset by both
classical machine learning models and state-of-the-art pre-trained large
language models. The experimental results reveal that the support vector
machine achieved the best performance amongst the classical machine learning
models while _xlm-roberta base_ achieved the best performance amongst the
large language models.

Contributions: The main contributions of this paper are:

  1. 1.

We developed a manually annotated COVID-19 vaccination acceptance dataset in
Nigerian cyberspace.

  2. 2.

We analyzed the data and derived novel insights from the data using multiple
exploratory data analysis techniques like topic modelling.

  3. 3.

We benchmark the dataset on COVID-19 vaccination sentiment analysis using both
classical machine learning techniques and large language models.

The remainder of this paper is organized as follows: Section 2 summarises the
related works. Section 3 provides a detailed description of the research
methodology. Section 4 presents the results and discussion. Finally, Section 5
concludes the paper with recommendations and future works.

##  II Related work

#### COVID-19 Vaccine Sentiment Analysis:

People’s opinion towards the COVID-19 vaccine has been a subject of study in
the research community and sentiment analysis is among the most widely used
approaches for this task [8, 9, 10]. A study by [11] collected and analysed
tweets on COVID-19 to understand and extract the most discussed topics. They
found "Vaccination programme" to be the widely discussed topic while [12]
conducted a similar research and found "vaccine hesitancy", "vaccine
acceptance" and "vaccine safety" as the most discussed topic related to the
COVID-19 vaccine. The study in [3] used social media data to analyse people’s
opinions towards the acceptance of COVID-19 vaccines in the US. The vaccine
acceptance index (VAI) was used to determine how well people accept these
vaccines. The result of their experiment shows a positive increase in the
acceptance of the COVID-19 vaccines in the last quarter of 2021. This result
contradicts the findings of [13] that says most people are not willing to
administer the vaccine. A related study by [14] analysed tweets related to
people’s acceptance of the Pfizer vaccine and found that most people have
negative perceptions of the vaccine.

#### Fine-tuning Large Language Models:

Pre-trained language models are trained on a large amount of data which makes
them very popular in NLP tasks. Researchers now prefer fine-tuning instead of
training a model from scratch, especially for small datasets [15]. Fine-tuning
entails changing the weights of these pre-trained models on a new task to
achieve better performance [16]. Keshavarzi et al. [17] fine-tuned BERT [18]
to investigate misinformation on Covid-19 vaccine from tweets. Similarly, [19]
used BERT to predict COVID-19 vaccine’s side effects using data from social
media. The study in [20] fine-tuned RoBERTa [21] to classify sentiments in
COVID-19 vaccine tweets.

##  III Research Methodology

###  III-A Data collection and Annotation

We collected tweets from Twitter (now X) using the Twitter Academic API from
1st November 2020 to 30th June 2022. We collected tweets keywords relating to
COVID-19 vaccination using keywords like vaccin*, moderna, pfizer, biontech,
gamaleya, janssen, astrazeneca, astrazeneca, covishield, sinopharm, jab,
microchips, infertility, tracking, magnet. The collection was restricted to
only Tweets in Nigeria using location ID in the query. A total of 5200 tweets
were collected. After removing URLS, tweets with less than three words, and
tweets not in Latin characters, the tweets were reduced to 4320.

The tweets were then annotated by three (3) independent annotators using an
annotation guideline. The classes for the annotation are:

  1. 1.

Positive: If the user’s tweet implies a positive attitude or opinion towards
the vaccine (directly or indirectly). For example, the tweets contain positive
words or some terms that imply a positive attitude towards COVID-19 vaccine
like save lives, protect, appreciative, thankful, excited, or optimistic.

  2. 2.

Negative: If the user’s tweet implies a negative attitude or opinion towards
the vaccine. For example, the tweets contain negative words or some terms that
imply a negative attitude towards COVID-19 vaccine like kill, harmful,
tracker, chip, critical, angry, disappointed in, pessimistic, expressing
sarcasm about, or mocking COVID-19 vaccine.

  3. 3.

Neutral: If the user’s tweet does not imply any opinion.

  4. 4.

Indeterminate: If the tweet is unintelligible.

After the annotation, _majority vote_ was used to assign the final class of
tweets in the following manner:

  1. 1.

If all the annotators agree on the same class, then that class is
automatically accepted.

  2. 2.

If two annotators agree on either positive or negative and the last annotator
classifies the tweet as neutral, then the tweet is assigned the majority
class.

  3. 3.

If two annotators classify a tweet as positive and the third annotator
classifies it as negative, then the tweet is adjudicated by a new annotator.
This is also the case if two annotators classify a tweet as negative and the
third annotator classifies the tweet as positive.

  4. 4.

If all the annotators label the tweet differently, the tweet is discarded.

At the end of the annotation, 4,320 tweets were annotated. Table I gives an
example of tweets in the dataset and their annotated classes.

TABLE I:  Example of tweets and their annotated classes  SN | Tweet | Class
---|---|---
1 |  i took my COVID vaccine jab today monday at the flag off of the fct vaccination campaign i believe it is safe praise god |  Positive
2 |  taking a vaccine that was developed under a year goes against every damn thing we know about drug discovery and vaccine creation refer back to the vaccines are not effective if they were why are vaccinated people being asked to wear masks |  Negative
3 |  COVID vaccine is safe and effective ive gotten mine go and get yours |  Positve
4 |  so who didnt approve the madagascarn herb but the astrazeneca vaccine they approved is now giving issues all over the world oh sorry like this was what naija imported hmmmm |  Negative
5 |  gov makinde ssg others vaccinated as oyo govt kicks off COVID vaccination |  Neutral

###  III-B Exploratory Data Analysis (EDA)

Our approach is focused on analyzing public sentiment on Twitter towards
COVID-19 vaccines. The initial step we explored is Exploratory Data Analysis
(EDA), aimed at gaining valuable insights from the tweets. EDA entails the
examination and interpretation of information and relevant data to draw
conclusions regarding people’s opinions. Therefore, we visualized the most
common words used in COVID-19 discussions using the word clouds. Additionally,
we used bar chat to compare the values of positive, negative and neutral
tweets in our dataset. We also visualize the people’s sentiments towards the
most common COVID-19 vaccines.

###  III-C Models

For the prediction of people’s sentiment towards COVID-19 vaccine, we utilize
two categories of models. First, we model our prediction based on classical
machine learning algorithms. We used classical machine learning algorithms
because our dataset is unlikely to yield reasonable results with deep learning
models as they require a large training dataset. Additionally, classical
machine learning algorithms sometimes yield competitive results compared to
deep learning models despite being less computationally expensive. The machine
learning algorithms used are Support Vector Machines (SVM), Logistic
Regression, Naive Bayes, K-Nearest Neighbours (KNN) and Decision Trees.

Secondly, we model our prediction by fine-tuning large language models. Fine-
tuning is an effective method of utilizing pre-trained large language models
for downstream NLP tasks using available datasets. The large language models
we fine-tuned are Bert, Afro-XMLR, and XLM-roBERTa.

##  IV Results and Discussion

###  IV-A Findings from Exploratory Data Analysis

fig. 1 illustrates the findings from the EDA. Figure 1(a) shows the sentiment
distribution of people towards COVID-19 vaccines. This information helps gauge
whether Nigerians have embraced or rejected the safety of the vaccines. The
results show that more than 27% of the tweets carry positive sentiment towards
the vaccines, while 18% of the tweets carry negative sentiment. Therefore, we
can deduce that based on Twitter discussions, the majority of Nigerians either
believe that vaccines are secure and life-saving or hold a neutral stance on
the vaccines, while a smaller fraction of Nigerians have expressed concerns
about the vaccine’s safety.

(a) Percentage of positive, negative, and neutral tweets

(b) Sentiments of vaccine common brands

(c) Word cloud for positive tweets

(d) Word cloud for negative tweets

Figure 1: Exploratory data analysis

In fig. 1(b), we visualize people’s stance towards the most common COVID-19
vaccines, AstraZeneca, Pfizer, Moderna, and Johnson & Johnson. It can be
observed that the majority of the people hold a neutral stance on the brands,
notwithstanding, a small percentage of the people still hold a negative stance
towards the brands. Johnson & Johnson vaccine has the best stance with less
than 5% of the tweets having a negative sentiment.

Figures 1(c), LABEL: and 1(d) show the most common words people use in
conveying their opinion towards COVID-19 vaccine. It can be observed that when
individuals hold a positive sentiment towards vaccines, they employ words such
as _safe, free, help, and protect_. These words suggest that vaccination has
been successful for a segment of the population who view the vaccine as a
significant lifesaving measure during challenging times. Conversely, some
individuals hold opposing views, citing instances where people died despite
being vaccinated. In such cases where individuals hold negative sentiments,
they use words like _risk, kill, pain, problem, and wrong_ to convey their
concerns.

###  IV-B Classical Machine Learning Models

Table II shows the performance of classical machine learning algorithms on
COVID-19 vaccine sentiment classification. SVM and Logistic Regression
achieved the best performance performance with an F1-score of $65\%$. However,
SVM outperformed the decision tree in terms of accuracy. Naive Bayes perform
poorly with an F1-score of $45\%$ and accuracy of $57\%$. Similarly, the
Decision tree achieved poor results with F1-score of $53\%$ and an accuracy of
$53\%$.

TABLE II:  Performance of Classical Machine Learning Models  Model | Accuracy | Precision | Recall | F1- Score
---|---|---|---|---
SVM | 0.66 | 0.65 | 0.66 | 0.65
Logistic Regression | 0.61 | 0.63 | 0.61 | 0.65
Naive Bayes | 0.57 | 0.65 | 0.57 | 0.45
K Nearest Neighbours | 0.62 | 0.61 | 0.62 | 0.61
Decision Trees | 0.53 | 0.53 | 0.53 | 0.53

###  IV-C Fine-tuned Large Language Models

Table III shows the evaluation results of fine-tuned LLMs. The models
exhibited better performance, which aligns with expectations. XLM-Roberta and
Afro-XLMR achieved the best with and F1-score of $71\%$ and $69\%$
respectively, and an accuracy of $71\%$. These two algorithms have been
trained on African languages and as such were expected to attain superior
performance due to the prevalence of Nigerian English code-mixed with other
Nigerian languages in the dataset. Notwithstanding, the bert model also
performed competitively with F1-score and accuracy of $70\%$. The overall
effectiveness of LLMs can be attributed to their extensive training using
diverse natural language texts. From our analysis of the results, it is
evident that fine-tuning a pre-trained LLM with an appropriate dataset can
yield competitive results, even if the LLM was not initially pre-trained on
the specific language of that dataset.

TABLE III:  Performance of Fine-tuned Large Language Models  Model | Accuracy | Precision | Recall | F1- Score
---|---|---|---|---
Bert base | 0.70 | 0.70 | 0.70 | 0.70
Afro-xmlr base | 0.71 | 0.70 | 0.71 | 0.69
xlm-roberta base | 0.71 | 0.71 | 0.71 | 0.71

##  V Conclusion and Recommendation

In this work, we developed and publicly released a manually annotated tweet
dataset for COVID-19 vaccination exploration in Nigerian cyberspace. The
dataset consists of 4320 annotated tweets. In addition, we employed multiple
exploratory data analysis techniques to derive insights from the dataset. We
found that most tweets indicate people generally have a neutral sentiment
towards the COVID-19 vaccine. Notwithstanding, a good number of people also
hold positive sentiments towards COVID-19 vaccines. We also found out that the
majority of the people in Nigeria have no significant preference for the
common vaccine types, even though the Moderna vaccine has slightly more
positive sentiments compared to the other vaccines.

In Addition to the dataset, we also investigate the efficiency of fine-tuning
LLMs for COVID-19 vaccine acceptance prediction in contrast to classical
machine learning algorithms using a small dataset. We found out that all the
machine learning algorithms performed below the fine-tuned LLMs. This finding
can be attributed to the fundamental differences between these two model
categories. While LLMs, predominantly based on transformer architectures have
been trained to grasp contextual information, classical machine learning
models primarily focus on predicting labels without considering contextual
nuances. Among the classical machine learning algorithms, SVM outperformed the
other algorithms in terms of macro average F1-score. This superiority can be
elucidated by the algorithm’s ability to capture non-linear relationships
within the data points.

Future work should dig further to fine-tune the hyperparameters and
preprocessing steps to obtain the best model performance. Moreover, the
limitation of our study is the relatively small dataset, comprising fewer than
four thousand samples, coupled with label imbalance. These factors may
constrain the performance of the fine-tuned models.

## Ethics Statement

This work does not raise any ethical concerns.

## Acknowledgements

This work was made possible by the mentorship program of the HausaNLP research
group.

## References

  * [1] J. Cui, Z. Wang, S.-B. Ho, and E. Cambria, “Survey on sentiment analysis: evolution of research methods and topics,” _Artificial Intelligence Review_ , pp. 1–42, 2023.
  * [2] A. Al-Hashedi, B. Al-Fuhaidi, A. M. Mohsen, Y. Ali, H. A. Gamal Al-Kaf, W. Al-Sorori, and N. Maqtary, “Ensemble classifiers for arabic sentiment analysis of social network (twitter data) towards covid-19-related conspiracy theories,” _Applied Computational Intelligence and Soft Computing_ , vol. 2022, pp. 1–10, 2022.
  * [3] L. Li, J. Zhou, Z. Ma, M. T. Bensi, M. A. Hall, and G. B. Baecher, “Dynamic assessment of the covid-19 vaccine acceptance leveraging social media data,” _Journal of Biomedical Informatics_ , vol. 129, p. 104054, 2022.
  * [4] G. Blanco and A. Lourenço, “Optimism and pessimism analysis using deep learning on covid-19 related twitter conversations,” _Information processing & management_, vol. 59, no. 3, p. 102918, 2022.
  * [5] H. Yin, X. Song, S. Yang, and J. Li, “Sentiment analysis and topic modeling for covid-19 vaccine discussions,” _World Wide Web_ , vol. 25, no. 3, pp. 1067–1083, 2022.
  * [6] N. A. Obi-Ani, C. Anikwenze, and M. C. Isiani, “Social media and the covid-19 pandemic: Observations from nigeria,” _Cogent arts & humanities_, vol. 7, no. 1, p. 1799483, 2020.
  * [7] H. Adamu, S. L. Lutfi, N. H. A. H. Malim, R. Hassan, A. Di Vaio, and A. S. A. Mohamed, “Framing twitter public sentiment on nigerian government covid-19 palliatives distribution using machine learning,” _Sustainability_ , vol. 13, no. 6, p. 3497, 2021.
  * [8] I. Aygün, B. Kaya, and M. Kaya, “Aspect based twitter sentiment analysis on vaccination and vaccine types in covid-19 pandemic with deep learning,” _IEEE Journal of Biomedical and Health Informatics_ , vol. 26, no. 5, pp. 2360–2369, 2021.
  * [9] C. A. Melton, O. A. Olusanya, N. Ammar, and A. Shaban-Nejad, “Public sentiment analysis and topic modeling regarding covid-19 vaccines on the reddit social media platform: A call to action for strengthening vaccine confidence,” _Journal of Infection and Public Health_ , vol. 14, no. 10, pp. 1505–1512, 2021.
  * [10] A. R. Rahmanti, C.-H. Chien, A. A. Nursetyo, A. Husnayain, B. S. Wiratama, A. Fuad, H.-C. Yang, and Y.-C. J. Li, “Social media sentiment analysis to monitor the performance of vaccination coverage during the early phase of the national covid-19 vaccine rollout,” _Computer Methods and Programs in Biomedicine_ , vol. 221, p. 106838, 2022.
  * [11] T. Baldha, M. Mungalpara, P. Goradia, and S. Bharti, “Covid-19 vaccine tweets sentiment analysis and topic modelling for public opinion mining,” in _2021 International Conference on Artificial Intelligence and Machine Vision (AIMV)_. IEEE, 2021, pp. 1–6.
  * [12] A. Sarirete, “A bibliometric analysis of covid-19 vaccines and sentiment analysis,” _Procedia Computer Science_ , vol. 194, pp. 280–287, 2021.
  * [13] A. Fridman, R. Gershon, and A. Gneezy, “Covid-19 and vaccine hesitancy: A longitudinal study,” _PloS one_ , vol. 16, no. 4, p. e0250123, 2021.
  * [14] M. Shah, A. Vasant, K. Gandhi, and S. Panesar, “Sentiment analysis of twitter tweets for covid-19 pfizer vaccines,” in _2023 7th International Conference on Intelligent Computing and Control Systems (ICICCS)_. IEEE, 2023, pp. 929–937.
  * [15] A. M. Schoene, J. Ortega, S. Amir, and K. Church, “An example of (too much) hyper-parameter tuning in suicide ideation detection,” in _Proceedings of the International AAAI Conference on Web and Social Media_ , vol. 17, 2023, pp. 1158–1162.
  * [16] H. Jiang, P. He, W. Chen, X. Liu, J. Gao, and T. Zhao, “Smart: Robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization,” _arXiv preprint arXiv:1911.03437_ , 2019.
  * [17] A. Keshavarzi Arshadi, J. Webb, M. Salem, E. Cruz, S. Calad-Thomson, N. Ghadirian, J. Collins, E. Diez-Cecilia, B. Kelly, H. Goodarzi _et al._ , “Artificial intelligence for covid-19 drug discovery and vaccine development,” _Frontiers in Artificial Intelligence_ , p. 65, 2020.
  * [18] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [19] S. Cheon, T. Methiyothin, and I. Ahn, “Analysis of covid-19 vaccine adverse event using language model and unsupervised machine learning,” _Plos one_ , vol. 18, no. 2, p. e0282119, 2023.
  * [20] J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le, “Finetuned language models are zero-shot learners,” _arXiv preprint arXiv:2109.01652_ , 2021.
  * [21] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” _arXiv preprint arXiv:1907.11692_ , 2019.