2021

[1]Zheng Enhui

[1]Unmanned System Application Technology Research Institute, China Jiliang
University, Baiyang Street, Hangzhou, 310018, Zhejiang, China

2]R&D Department, Aerospace CH UAV Co., Ltd, Haihao Street, Taizhou, 318014,
Zhejiang, China

# Vision-Based UAV Localization System in Denial Environments

Dai Ming<EMAIL_ADDRESS>Huang Jinglin<EMAIL_ADDRESS>Zhuang Jiedong<EMAIL_ADDRESS>Lan Wenbo<EMAIL_ADDRESS>Cai
Yongheng<EMAIL_ADDRESS><EMAIL_ADDRESS>*  [

###### Abstract

Unmanned Aerial Vehicle (UAV) localization capability is critical in a Global
Navigation Satellite System (GNSS) denial environment. The aim of this paper
is to investigate the problem of locating the UAV itself through a purely
visual approach. This task mainly refers to: matching the corresponding geo-
tagged satellite images through the images acquired by the camera when the UAV
does not acquire GNSS signals, where the satellite images are the bridge
between the UAV images and the location information. However, the sampling
points of previous cross-view datasets based on UAVs are discrete in spatial
distribution and the inter-class relationships are not established. In the
actual process of UAV-localization, the inter-class feature similarity of the
proximity position distribution should be small due to the continuity of UAV
movement in space. In view of this, this paper have reformulated an intensive
dataset for UAV positioning tasks, which is named DenseUAV, aiming to solve
the problems caused by spatial distance and scale transformation in practical
application scenarios, so as to achieve high-precision UAV-localization in
GNSS denial environment. In addition, a new continuum-type evaluation metric
named SDM is designed to evaluate the accuracy of model matching by exploiting
the continuum of UAVs in space. Specifically, with the ideas of siamese
networks and metric learning, a transformer-based baseline was constructed to
enhance the capture of spatially subtle features. Ultimately, a neighbor-
search post-processing strategy was proposed to solve the problem of large
distance localisation bias.

###### keywords:

Geo-Localization, Transformer, UAV, Deep Learning

Figure 1: (a) depicts the criteria for judging the previous evaluation
metrics. (b) describes the evaluation criteria of our proposed SDM evaluation
metric where $d_{1-K}$ represents the spatial Euclidean distance between the
Drone-View Image and the most similar $1-K$ satellite images, and the
calculation process is shown in Equation 4, where $X$ represents the longitude
and $Y$ represents the latitude.$w_{1-K}$ are weighting factors and $s$ is an
amplification factor,set to $5\times{10^{3}}$ in this paper. (c) Compared the
evaluation process of R@1 and SDM, for a single sample, the evaluation
criterion of R@1 is discrete, i.e. 1 vs. 0. However, for the SDM evaluation
metric, the score varies with the spatial distance from ground-truth, with the
closer it is the higher the score.

##  1 Introduction

In the usual scenario, UAV localization relied on satellite systems to provide
accurate latitude and longitude position information. However, this approach
need to presuppose that satellite systems operate properly and that GNSS
signals are not interfered with or blocked. Once the UAV does not acquire GPS
signals, the UAV will not be able to operate properly. Therefore, the ability
to locate in denial environment is particularly important. In today’s world,
UAV has played a more and more pivotal role, and have been used in
applications such as military combat, agricultural operations, ground
reconnaissance and even civilian aerial photography (Zheng et al, 2020a; Hu
and Lee, 2020; Middelberg et al, 2014; McManus et al, 2014). With the
improvement of on-board device performance and the lightweight of hardware,
many algorithms such as object tracking, object detection, SLAM and other
algorithms (Yu et al, 2020; Ma et al, 2021) have been applied to UAV and
generated great social benefits.

The University-1652 (Zheng et al, 2020a) dataset firstly introduces the drone-
view to cross-view geo-localization, which regarded drone-view as an
intermediate transition view to alleviates the difficulty between matching
through ground-view images and satellite-view images. Meanwhile, they have
also proposed drone-view target localization and drone navigation tasks to
extend the application of drone in geo-localization. As a successor, we still
explore the subject of drones. The difference, however, is that this paper
aims at solving the problem of locating the UAV itself and the implementation
is broadly consistent with the geo-localization task, in which the drone-view
image is used to retrieve the most similar geo-tagged satellite image in the
satellite database to obtain the current location information of the UAV. In a
broad sense, The University-1652 locates the building in view, while we locate
the device that captures the object such as a camera, whose positioning goal
is in the same vein as CVUSA (Zhai et al, 2017a) and CVACT (Liu and Li,
2019a). The difference is that our approach does not use panoramic images as
the images of the drone-view, otherwise we use a simpler solution, i.e., the
UAV camera is vertical ground down to solve the view deviation problem, which
is an advantage that the ground camera does not have. Another notable feature
is that our dataset is densely collected, which means that there is an overlap
between adjacent images, which was designed to increase the influence of
spatial position deviation and thus achieve high accuracy positioning.

CMC (Zhao et al, 2013; Zheng et al, 2015) and mAP are the mainstream
evaluation metrics for image retrieval tasks. They are based on classification
to judge accuracy and are very effective when no spatial relationships are
introduced, however, when assessing spatial positioning accuracy, we found
some significant drawbacks in these evaluation metrics. In the process of UAV
positioning, we expect the matched satellite map to be as close to the actual
location as possible. First of all, it should be noted that in the inference
stage, the satellite gallery database must be dense, which means that the same
prominent landmarks (such as buildings, lakes, trees, etc.) will appear in
multiple satellite images. We have made an extreme example in Figure 1(a),
where the red box is a satellite image that corresponds exactly to the
geographic location of the drone-view image, and others in blue box are
satellite-view images with a only small position shift. If the CMC and mAP
evaluation metrics are used to judge, only the image in the red box is the
True-Matched sample and the images in blue box are all False-Matched samples,
which is obviously not in line with the actual application of UAV positioning.
Based on this, we propose the evaluation metric SDM to change the previous
pattern of 0 or 1 by assigning the corresponding score according to the
matching distance, the closer the matching position is to the target, the
higher the score should be. This design approach is mainly based on our actual
application scenarios, where UAVs inevitably have matching errors during the
matching process, and a slight offset is an acceptable phenomenon, while an
excessive offset will largely undermine the reliability of subsequent
positioning.

The development of transformer in computer vision is in full swing. Compared
to the pure CNN approach, transformer shows significant advantages with its
self-attention mechanism. Taking a closer look at the CNN-based approach, we
mainly found two potential problems. (I) Vision-based geo-localization needs
to dig out the relevant information between contexts. Images from different
domains have positional transformations such as rotation, scale and offset.
Therefore, fully understanding the semantic information of the global context
is necessary. However, CNN-based methods mainly focus on small discriminative
regions due to Gaussian distribution of effective receptive fields (Vaswani et
al, 2017). Given the limitations of the pure CNN-based methods, the attention
modules have been introduced to explore long-range relationships (Wang et al,
2018). However, most of the methods embed the attention mechanism into the
deep convolutional network, which enhances contextual connections to a certain
extent. (II) Fine-grained information is very important for the task of
retrieval. The down-sampling operations i.e., pooling and stride convolution
of the CNN-based method can reduce the resolution of the image, while
inevitably destroying the recognizable fine-grained information. In this
paper, we propose a Transformer baseline for the task of UAV vision
localization. Similar to the cross-view ground-aerial matching task, we use
siamese network architecture with shared weights, a fusion of metric learning
and representation learning to learn intra-class similarity and inter-class
differences. Conventionally, we use feature vectors to characterize the images
of drone-view and satellite-view, and euclidean distance to measure the gap
between features representation.

Figure 2: Diagram of the flow structure of a UAV positioning mission during
the execution phase. localization Strategies refer to post-processing
operations after model inference, such as Global-Search and Neighbor-Search.
blue circles in localization Result (Part) are real UAV locations, red are
matched locations, and numbers indicate correspondence.

Our benchmark provides an end-to-end UAV-localization solution. The complete
process is shown in Figure 2. (1) Acquire satellite images from Google Earth
with 20-level and perform isometric multi-scale cutting of the satellite maps
to build satellite database. (2) The images in satellite database are then
passed through our geo-localization network to obtain the corresponding
feature representations, and the UAV images undergo the same operation
mentioned above. (3) Compute the feature distance between the drone-view image
and every image in satellite databases. In order to visualize the process of
localization, we use the GPS information that comes from the UAV to do the
verification (in the real scenario the UAV does not have GPS, so this is just
for visualization). (4) The localization strategies part is used to control
the scope of the search space (global search or neighbor-search). (5) Then we
can retrieve the satellite images with the closest distance to the drone-view
image from the satellite database, and the top1-10 images are chosen to show
in this paper. Finally, we visualize some images taken by UAV in the local map
(using top1 satellite image as the matching term).

In the process of actual application, we found that the UAV positioning system
based on global searching still has many cases of incorrect matching. One of
the most important reasons is that the satellite map is so large that the
images cut out from the map are somewhat very similar. Coupled with the fact
that the satellite images themselves are of low clarity and there will also be
areas covered by shadows. Thus, the probability of incorrect matching will be
greatly increased. An efficient solution to alleviate the above problems is to
reduce the search range. Relying on the continuity of object motion in space,
we adopt the neighbor-serach strategy to avoid some images with high
similarity but far away from each other. In the implementation, we construct a
circle with a constant distance based on the GPS information obtained from the
previous match, the radius of the circle is also determined by the flight
speed and positioning sampling frequency of the UAV. Using this circle as the
search domain, the satellite images within the circle are filtered to
construct a new satellite database as the candidate gallery library. This
approach not only improves the accuracy but also the speed of retrieval.
However, a hyper-parameter $M$ will be introduced and this hyper-parameter
needs to be adjusted in combination with the UAV related parameters during the
flight of the UAV to achieve better performance.

The contributions in this paper are as follows.

  * $\bullet$

we propose a dense UAV-based geo-localization dataset, named DenseUAV, which
aims to achieve high-precision positioning of drones.

  * $\bullet$

We propose a new evaluation criterion, SDM, to assess the accuracy of UAV
positioning based on the deviation of spatial position.

  * $\bullet$

Based on the proposed DenseUAV dataset, we constructed a transformer-based
baseline to accomplish the UAV localization task. Meanwhile, an improved
neighbor-search scheme is proposed to further enhance the reliability of UAV
visual localization in the denial environment.

##  2 Related Work

###  2.1 Geo-Localization Dataset

Initially, the geo-localization task was proposed to solve the ground-to-
aerial matching problem, where ground refers to images captured by users using
mobile cameras and aerial refers to images captured by satellites. One of the
earliest works (Lin et al, 2015) proposed to leverage the public sources to
build image pairs for the ground-view and aerial-view images, which consists
of 78k image pairs from two views, i.e., $45^{o}$ bird view and ground view.
Subsequently, (Tian et al, 2017) collected image pairs from the town view for
the task of geo-localization, which argued that architecture can provide more
notable features for geo-localization. (Vo and Hays, 2016) aimed to determine
the location and orientation of a ground-level query image by matching a
reference database of satellite images, and they collected a dataset with one
million pairs of street view and overhead images sampled from eleven U.S.
cities. Later, two well-known large-scale datasets i.e., CVUSA (Zhai et al,
2017a) and CVACT (Liu and Li, 2019a) made grand entrance. (Zhai et al, 2017a)
constructed image pairs from ground-based panoramic images and satellite
images, while (Liu and Li, 2019a) added spatial factors to CVUSA, i.e.,
orientation maps. The latter three mentioned datasets targeted users with
photographic devices, while the first two mentioned datasets targeted the
buildings in images. Recently, University-1652 (Zheng et al, 2020a) introduced
the drone-view into cross-view geo-localization, using the drone-view as a
transition view to reduce the difficulty of the matching between ground-view
and satellite-view, and regarded the cross-view geo-localization as a
retrieval task. In addition, University-1652 proposed two UAV-based subtasks
respectively called drone-view target localization and drone navigation, which
drived the application of UAVs for cross-view geolocation. The emergence of
DenseUAV was inspired by University-1652, the different is that
University-1652 focuses on the recognition of buildings and selects buildings
with certain features and styles, and the space distribution of these
buildings is discrete. And it aims to solves the problem of localization of
buildings in images. In contract, DenseUAV is designed to solve the problem of
the localization of UAV itself under the condition of GPS-free.

###  2.2 Deeply-Learned Feature For geo-localization

Most prior work has viewed geo-localization as an image retrieval task. The
most significant characteristics are the bias in perspective between the
matched images and the different sources of the images. Thus a critical factor
for cross-view geo-localization to determine retrieval accuracy is the ability
to overcome the viewpoint-invariant problem between different domains (Long et
al, 2017), such as the existence of a 90-degree span between ground and
satellite views, where the ground view can result in many objects visible in
the satellite view not appearing in the ground view due to factors such as
object occlusion.

One line focus on metric learning and improves the ability of the model to
discriminate differentiation by optimizing the loss function. Triplet loss
(Liu et al, 2017) was proposed to improve the reliability of matching by
closing the distance between features of the positive pairs and distancing the
distance between features of negative pairs. The contrastive loss, pulling the
distance between positive pairs, could further improve the geo-localization
results (Hu and Lee, 2020; Workman et al, 2015). (Zheng et al, 2020b) applied
instance loss and verification loss together to optimize the network, and
achieve competitive results. (Hu et al, 2018) proposed a weighted soft margin
ranking loss, which not only speeds up the training convergence but also
improves the retrieval accuracy. (Liu and Li, 2019b) proposed a BNNeck to
improve the coordination of ID loss and triplet loss. (Sun et al, 2020)
proposed a unified framework to optimize classification learning and pairwise
learning.

Another line focus on the spatial information asymmetry between ground-to-
aerial matching. (Zhai et al, 2017b) proposed an end-to-end learning approach
to minimize the difference between the semantic segmentation extracted
directly from the ground image and the semantic segmentation predicted solely
based on the aerial image. (Arandjelovic et al, 2016) inserted the NetVLAD
layer to express the relationship between features and each cluster in terms
of probability values and extract discriminative features. Further, (Liu and
Li, 2019b) designed a Siamese network which explicitly encodes the orientation
(i.e., spherical directions) of each pixel of the images. Similarly, (Shi et
al, 2020b) firstly applied a regular polar transform to warp an aerial image
to bring its domain closer to the ground-view panoramic and then add a
subsequent spatial-attention mechanism to improve the robustness of feature
representation. (Shi et al, 2020a) addressed the cross-view domain gap by
applying a polar transform to the aerial images to approximately align the
images up to an unknown azimuth angle. (Toker et al, 2021) proposed a GAN
structure to create realistic street views from satellite images and localizes
the corresponding query street-view simultaneously in an end-to-end manner.
(Wang et al, 2021) proposed a square chunking strategy to fully extract useful
information from the edges, and the chunking scheme results in a significant
improvement in localization performance.

###  2.3 Transformer In Vision

The attention mechanism (Vaswani et al, 2017) of the transformer model was
first proposed to solve problems in the field of Natural Language Processing.
recently, (Dosovitskiy et al, 2020) firstly applied the transformer model to
the task of image classification. Subsequently, Transformer has achieved
state-of-the-art performance in various domains of vision. DETR (Carion et al,
2020) was the first object detection framework that successfully integrates
the transformer as the central building block of the detection pipeline. SETR
(Zheng et al, 2021) treated semantic segmentation as a sequence-to-sequence
prediction task through a pure transformer. TransGAN (Jiang et al, 2021) built
a generator and a discriminator based on two transformer structures. TTSR
(Yang et al, 2020) restored the texture information of the image super-
resolution result based on the transformer. TransReID (He et al, 2021) applied
the transformer to the field of retrieval for the first time and achieved
similar results with the CNN-based method.

Also there is a lot of work using Transformer as the backbone network to
extract features, DeIT (Touvron et al, 2021) introduces a teacher-student
strategy specific for transformers to speed up Vit training without the
requirement of large-scale pretraining data. Swin Transformer (Liu et al,
2021) proposed a hierarchical Transformer whose representation is computed
with Shifted windows. MobileVit (Mehta and Rastegari, 2021) combine the
strengths of CNN and Vit to build a lightweight and low latency network for
mobile vision tasks.

Table 1: geo-localization dataset information summary table

Datasets | DenseUAV | University-1652 | CVUSA | CVACT | Lin et al. | Tian et al. | Vo et al.
---|---|---|---|---|---|---|---
#training | 2256 $\times{}$ 6 | 701$\times{}$71.64 | 35.5k$\times{}$2 | 35.5k$\times{}$2 | 37.5k$\times{}$2 | 15.7k$\times{}$2 | 900k$\times{}$2
Platform | Drone,Satellite | Drone,Ground,Satellite | Ground,Satellite | Ground,Satellite | Ground,Aerial | Ground,Aerial | Ground,Satellite
#imgs./location | 3 + 3 | 54 + 16.64 + 1 | 1 + 1 | 1 + 1 | 1 + 1 | 1 + 1 | g 1 + 1
Target | UAV | Building | User | User | Building | Building | User
Evaluation | SDM | Recall@K & AP | Recall@K | Recall@K | PR curves & AP | PR curves & AP | Recall@K

Table 2: DenseUAV data volume composition table. Where #classes refers to the
number of categories and #universities refers to the number of Universities we
sampled.

split | #imgs | #classes | #universities
---|---|---|---
drone | satellite
Training | 6768 | 6768 | 2256 | 10
Query | 2331 | 2331 | 777 | 4
Gallery | 9099 | 9099 | 3033 | 14

Figure 3: A diagram describing the process of creating the dataset. The red
circles represent the sampling locations, with the sampling points spaced 20
metres apart and sampled at 80, 90 and 100 metres respectively. The green
arrows represent the orientation of the camera.

##  3 Methodology

###  3.1 Dataset Description

In this paper, we propose a dense UAV-based geo-localization dataset, which
differs from previous cross-view geo-localization datasets mainly in the term
“dense”. “dense” refers to the fact that the drone-view images in the training
phase are dense in spatial distribution, in addition to the fact that the
database of satellite to be matched in the testing phase is also highly dense
in spatial distribution. The ultimate goal of the proposed DenseUAV is to
achieve high accuracy positioning of UAV. To our best knowledge, Another
difference is that the DensUAV regards the UAV as the localization target,
which has not been proposed and addressed by researchers in the past. As shown
in Table 1, we enumerate many elements of the previously proposed dataset for
further comparison.

The following three main factors were taken into account in the way of dataset
acquisition. First, considering the influence of the height of the UAV flight
on the image scale. The UAV images we collected consisted of 3 different
heights, 80, 90, and 100 meters (relative to the ground), as shown in Figure
3. The sampling process needs to ensure that the latitude and longitude of the
images sampled at 3 different heights are almost identical. We applied the
waypoint action of DJI Plot to achieve this and control the error within 1m.
Second, considering the influence of weather and light level on the image
quality, we adopt the random weather and random time period sampling method.
The random weather mainly includes sunny and cloudy days, and the random time
period mainly refers to the random sampling between 6:00 am and 6:00 pm.
Third, taking into account the perspective bias caused by the change of UAV
camera orientation, we unified the UAV camera with the vertical ground facing
downward. In addition, considering the problem of UAV flight orientation, this
problem is solved by a kind of rotational data enhancement, which is
introduced in Section 4.2. In addition, we set the distance between each
sampling point to 20m. There are two purposes for this. On the one hand, we
standardize our dataset by this equally spaced sampling, on the other hand,
the dataset constructed by dense sampling can enhanced ability of model to
learn subtle features.

For the composition of the dataset, we selected 14 universities in Hangzhou,
Zhejiang, China as our sampling area due to the large labor and hardware costs
for producing the dataset. As shown in Table 2, among them, 6768 drone-view
images from 2256 sampling points of 10 universities and an equal amount of
satellite images are used for training. In the test set, the query set
contains 2331 drone-view images from 777 sampling points of 4 universities and
an equal number of satellite images, and the gallery set contains a total of
18198 images from 3033 sampling points of all 14 universities.

###  3.2 Evaluation Indicators

With recurring content between images in dense datasets and a dense
distribution of categories, commonly used evaluation metrics such as Cumulated
Matching Characteristics (CMC) curve (Zhao et al, 2013; Zheng et al, 2015) and
mean Average Precision (mAP) will no longer be suitable for assessing the
accuracy of drone positioning. In the following, we do some specific analysis
to explain the reasons for this.

mAP and CMC mainly consider whether the classification is accurate or not.
Specifically, these evaluation criteria are both judging whether the selected
image in the gallery belongs to the same category as the image of query. In
terms of a single sample, these evaluation metrics simply care whether it is 0
or 1, which we consider as a discrete evaluation indicator. Take rank1 in CMC
evaluation index as an example, its expression is shown in Equation 1 and
Equation 2.

$I(l_{q},l_{i})=\left\\{\begin{array}[]{l}1\ \ \ \ \ if\ \ l_{q}=l_{i}\\\ 0\ \
\ \ \ if\ \ l_{q}\not=l_{i}\end{array}\right.\ \ \ $ (1)

Where $l_{q}$ corresponds to the category of query, and $l_{i}$ corresponds to
the category corresponding to the $ith$ image sorted from smallest to largest
after calculating the Euclidean distance and ascending the order, which is 1
if it belongs to the same category and 0 if it is not. And the rank1
expression is as follows.

$rank1=\ \frac{1}{\left\|{}Q\right\|{}}\sum_{q\in{}Q}I(l_{q},l_{1})$ (2)

Where $Q$ is the set of all query images and $\|{}Q\|{}$ denotes the number of
images in the set Q. It can be seen that the rank1 increases when and only
when the category of query and the category of the closest image in the
gallery are the same, otherwise they are all regarded as False-Matched
samples.

However, there is a characteristic in the actual operation of the UAV that the
satellite images are spatially continuous and dense, and the gap of feature
information contained between adjacent images is small. As shown in Figure 1
(a), if the previous evaluation criteria of CMC and mAP is used, only the
satellite images in the red dashed part are correct, $I(l_{q},l_{1})=1$, and
the satellite images in the blue dashed part are wrong, $I(l_{q},l_{1})=0$,
although the deviation is very small. In the actual UAV localization task, the
UAV is expected to position itself as closer to the real location as possible.
Minor positioning deviations is inevitable, however the presence of large
deviations is not expected, and the CMC and mAP metrics do not express this
difference in spatial distance. In order to increase the judgment of the
spatial distance of the UAV, the true GPS coordinate positions of all
satellite images and UAV images were recorded. Based on this, an evaluation
metric for measuring spatial distance (SDM) is proposed as an evaluation
criterion for UAV localizatioin task. Its expression for a single query sample
is as follows.

${SDM}_{K}=\sum_{i=1}^{K}\frac{(K-i+1)}{e^{{s\times{}d}_{i}}}$ (3)

$d_{i}=\sqrt{{(x_{q}-x_{i})}^{2}+{(y_{q}-y_{i})}^{2}}$ (4)

where the numerator $(K-i+1)$ is the weight $w$, which is presented in Figure
1(b). The weight w is assigned based on the feature distance, and a greater
weight is set for the gallery image that is closer to the query feature. $K$
denotes the top $K$ samples in the gallery that are closest to the distance of
the feature vector calculated by query. $x_{q}$ and $x_{i}$ denote the
longitude corresponding to the query and gallery images, respectively, while
$y_{q}$ and $y_{i}$ denote the latitude. In short, $d_{i}$ represents the
spatial Euclidean distance between two images. $S$ is a scaling factor, and in
this paper, $S$ is set to $5\times{10^{3}}$.

The weights of the SDM are then normalised to give the full formula for the
SDM as follows.

${SDM}_{K}=\frac{\sum_{i=1}^{K}\frac{(K-i+1)}{e^{{s\times{}d}_{i}}}}{\sum_{i=1}^{K}(K-i+1)}$
(5)

The interval of SDM metric is distributed between 0 and 1, and the larger SDM
value indicates the better localization performance of the model. For a
clearer representation, the visualization process of the calculation is
presented in Figure 2(b). $d_{1-K}$ represents the spatial distance between
query and $rank_{1-K}$ images of the candidate frame.

To illustrate this difference more simply, we compare the SDM1 and R@1 methods
in Figure 1(c), where our method is like a continuous evaluation criterion,
while R@1 just simply chooses between 0 and 1.

Figure 4: A diagram of the overall structure of our Baseline which contains
both training and testing phases. Classifier Layer contains linear layer,
relu, batchnorm1d and dropout. CE Loss represents cross-entropy loss without
label-smooth.

###  3.3 Benchmark Introduction

Based on the proposed baseline of University-1652, we use small size vision
transformer (VitS) as the backbone of the network to help the network fully
understand the feature relationships between different perspectives. As shown
in Table 3. the performance of the VitS-based baseline is much better than
that of the Resnet50-based baseline. this is due to the fact that in the task
of intensive UAV geo-localization, there are viewpoint biases between images
from different sources, making contextual correlation information crucial, and
fully understanding the long-range global feature information of images has a
significant effect, the shortcoming is that the VitS-based model is slightly
slower than the resnet50-based model in the inference process, and the
inference time is roughly 1.27 times that of the Resnet50.

The overall structure of our baseline is shown in Figure 4, where the input
image is cut into 16$\times$16 patches equally, and the information of each
patch is compressed by a Linear Projection. Which is composed of a convolution
with a kernel of 16$\times$16 and a stride of 16, and expand the number of
channels to 768. After that, the dimension is flattened to generate patches,
and Each patch plus a Position Embedding. In addition, to express the global
features we embed an additional embedding patch for the global feature
representation. After passing through the Transformer Layer (The VitS used in
this paper consists of a total of 8 self-attentive blocks, with the same
structure as TransReid (He et al, 2021)), we get the features with the same
dimension as the input, and the $cls\\_token$ is regarded as the final feature
representation. The output is divided into two parts, one of which converts
the 768-dimensional VitS output into a 512-dimensional feature vector via a
fully-connected layer to calculate Triplet Loss, and the other part goes
through a fully-connected layer on this 512-dimensional feature vector for
classification.

###  3.4 UAV Positioning Applications

Another contribution of this paper is that we have made a practical
application based on our benchmark. The overall process of UAV localization is
actually very simple, and the basic way is to retrieve the closest geo-tagged
satellite map from the satellite database through the UAV image, and then
obtain the GPS information of the location of the UAV at this time. However,
there are some post-processing methods that can simultaneously improve the
retrieval speed and accuracy, such as narrowing the retrieval range.

Vision-based UAV localization is a fresh approach that can be optimized in
terms of localization accuracy and time consumption through many post-
processing methods (Corresponds to the positioning strategy in Figure 2), and
in this paper we propose the global search method as well as its improved
version of the neighbor-search method. The global search method retrieves the
whole satellite database, which is downloaded in advance and cut with a
sliding window. In order to improve the accuracy of retrieval, this paper
adopts the cutting method of multi-scale and repeated area (multi-scale refers
to the different sizes of the satellite images, and repeated area refers to
the sliding distance of the sliding window is smaller than the window size,
and the sliding distance used in the testing process of this paper is 1/4 of
the sliding window size), and we have made a visual demonstration in Section
5.1. Although the baseline-based positioning accuracy has achieved good
performance, with the increase of the satellite database, the positioning
accuracy will be greatly reduced, and there will be a situation of matching to
a spatially distant location, producing a large positioning deviation, which
is very unfavorable for UAV positioning.

In order to improve the above drawbacks in the global search, we propose a
neighbor-search method to make up for the defects of the global search, which
uses the spatial motion continuity of the UAV, takes the previous matching
position of the UAV as the center of the circle and takes a suitable radius
$M$ to form a circle as the search domain, and the search domain $X$ can be
expressed as the following formula.

$X=\bigcup_{\theta{}\epsilon{}H}{(d}_{\theta{}}<M)$ (6)

where $H$ denotes the satellite database, $d_{\theta}$ denotes the distance
between the $\theta{th}$ satellite image and the current center position, and
$M$ is the search radius. The neighbor-serach method eliminates impractical
localisation results by reducing the area to be retrieved. However, it is
worth noting that the parameter $M$ is required to be adjusted according to
the flight speed and sampling retrieval frequency of the UAV. In general,
compared to the global search method, the neighbor-search method reduce the
scope of retrieval, which can not only reduce the amount of computation but
also improve the matching accuracy. However, due to the time difference
between the real acquired satellite images, which are historical images, and
the UAV images, and the poor clarity of satellite image, the task is still
very challenging.

Figure 5: Schematic diagram of a data enhancement method for enriching the
flight orientation information of a UAV. The red dashed line is the largest
interior square centred on the centre of the image, the green dashed line is
the largest interior circle of that square, and the yellow dashed line
represents a number of the largest interior squares of the circle. $\theta$
indicates the rotation angle.

##  4 Ablation Experiments

###  4.1 Implement Details

We used VitS pre-trained on the ImageNet dataset as the backbone, and added a
512-dimensional fully connected layer and a classification layer after
pooling, while removing the original classification branch. The model is
trained by stochastic gradient descent with the learning rate is 0.01 for the
new-added layers and 0.003 for the rest layers. dropout rate is 0.5. In the
training process, the input image size is 256$\times$256, and some data
augmentation methods are applied to expand the data, including color
enhancement, horizontal flipping and our proposed rotational method in 4.2. In
the testing phase we use the trained model to extract the features of the
images and calculate the Euclidean distance between the features to
characterize the similarity between the query and the candidate images in the
gallery. The final retrieval results are derived based on ranking of feature
distance. It is worth mentioning that for different views, we use only one
model for training, which means that all weights are shared.

Table 3: Table of experimental results for UAV flight direction data
augmentation. o denotes without our proposed data augmentation, w denotes
with.

Backbone | w/o rotate | SDM1 | SDM3 | SDM5 | SDM10
---|---|---|---|---|---
Resnet50 | o | 0.143 | 0.125 | 0.115 | 0.100
Resnet50 | w | 0.155 | 0.136 | 0.128 | 0.114
VitS | o | 0.454 | 0.436 | 0.390 | 0.305
VitS | w | 0.525 | 0.504 | 0.449 | 0.345

###  4.2 Flight direction data expansion

In the Vision-based UAV localization task, the UAV and satellite images are
supposed to correspond one-to-one, but such one-to-one correspondence is not
very helpful for model learning, and we expect to create more situations that
match real scenarios. Both the flight direction and the flight altitude of the
UAV are uncertainties in real scenarios. Intuitively, these two factors affect
the rotation and scale of the image, respectively. In terms of the scale
variation of the UAV images, we use the acquisition of samples by UAVs at
different altitudes in real scenes. On the uncertainty of UAV flight
direction, a rotate augmentation method consistent with realistic UAV sampling
is proposed to expand drone-view images.

The rotation augmentation method proposed in this paper does not rotate the
image directly in the original image, which does not change the information
contained in the image and does not match the actual UAV rotation scenario. As
shown in the top panel of Figure 5. The target of the crop is the largest
inline circle in the original image, and the largest inline square in that
circle is cut out according to the rotation angle and then using a perspective
transformation method to arrange the picture horizontally. $\theta$ refers to
the angle of rotation. This data augmentation not only expands the data in the
direction of the drone’s flight, but also enhances the generalisation of the
model by erasing the image information at the edges, which is akin to applying
some patches to the image, allowing the model to dig for more generic
features. Besides, erasing the edge information allows the model to focus more
on the information in the centre of the image, which is what we would expect
as the centre point is closer to the real position of the drone. Six randomly
rotated cropped images are shown on the bottom two rows of Figure 5. To verify
the effectiveness of our proposed rotation augmentation method for UAV
localization, we developed experiments for validation, as shown in Table 3,
after the rotational change, the SDM metrics rise significantly by points.

Figure 6: Graph of the effect of K on the SDM indicator Table 4: Table of test
results for different dataset training weights in DenseUAV. FeatureDim refers
to the dimensionality of the output feature vector.

Dataset | FeatureDim | SDM1 | SDM3 | SDM5 | SDM10
---|---|---|---|---|---
ImageNet | 2048 | 0.023 | 0.022 | 0.020 | 0.018
University-1652 | 512 | 0.046 | 0.042 | 0.038 | 0.033
DenseUAV | 512 | 0.717 | 0.695 | 0.615 | 0.467

###  4.3 Impact of K in SDM

$K$ in $SDM_{K}$ refers to the K samples that are closest to the feature
distance of query, and $SDM_{K}$ measures the spatial distance by these TopK
samples and ignore the rest. Thus K determines the number of candidate gallery
images that participate in the SDM calculation. To verify the effect of $K$ on
the SDM evaluation metric, we vary K from 1 to 100 to observe the trend of
SDM. As shown in Figure 6, the curve decreases rapidly when $K>3$, which is
due to the fact that each query has only 3 corresponding gallery images in our
dataset. in addition, $SDM_{K}$ shows a monotonically decreasing trend.

###  4.4 Non-Dense Features vs. Dense Features

We tested features from different types of datasets in the DenseUAV test set,
divided into Non-Dense and Dense types. (1) We used the VitS model pre-trained
in ImageNet as the standard model, with an output dimension of 2048. In
addition, we trained the VitS model in the University-1652 dataset, with an
output dimension of 512. (2) We also use the VitS model in the DenseUAV
dataset, and the dimension of the output feature vector is also 512. The
models were evaluated in the test set of DenseUAV using SDM evaluation
metrics. As shown in Table 4, the results indicate that the features trained
in ImageNet and University-1652 lack the ability to discriminate spatial
locations, which is also related to the fact that the dataset itself is not
targeted for dense tasks, which justifies the need for dense datasets to be
proposed.

Table 5: SDM accuracy for different backbone training in DenseUAV and
inference schedules for the testing phase.

Backbone | Inference Time | SDM1 | SDM3 | SDM5 | SDM10
---|---|---|---|---|---
ResNet-50 | 1x | 0.143 | 0.125 | 0.115 | 0.100
ResNet-101 | 1.25x | 0.212 | 0.194 | 0.179 | 0.153
ResNeSt-50 | 1.2x | 0.151 | 0.135 | 0.124 | 0.108
VitS | 1.27x | 0.454 | 0.436 | 0.390 | 0.305

###  4.5 Effect Of Backbone

To verify the impact of the backbone on the UAV localization task, we trained
some backbone i.e. ResNet-50, ResNet-101, ResNeSt-50 (Zhang et al, 2020) and
VitS on the DenseUAV dataset and tested them with SDM metrics. We also
evaluate the time consumption of different backbone networks during the
testing phase. All the backbone networks import the pre-training weights of
ImageNet and only apply CrossEntropy Loss for training. As shown in Table 5,
we found that the Transformer structure-based model had a stronger performance
in the UAV localization task compared to the CNN structure. In terms of
inference time cost, VitS has a test time of 1.27$\times$, which is about the
same as ResNet-101’s 1.25$\times$, but improves nearly 24 points in the SDM1
evaluation metric.

Table 6: Results of different state-of-the-art models tested in DenseUAV with
R@1, R@Top1%,mAP and SDM1 evaluation metrics.

Method | Feature Dim | R@1 | R@Top1% | mAP |  SDM1
---|---|---|---|---|---
University-baseline (Zheng et al, 2020a) | 512 | 7.51% | 56.2% | 7.57% | 0.155
Wehgited Soft Margin Triplet Loss (Liu and Li, 2019b) | 512 | 9.70% | 60.53% | 9.72% | 0.168
LPN (Wang et al, 2021) | 2048 | 35.26% | 85.33% | 31.03% | 0.415
MSBA (Zhuang et al, 2021) | 3072 | 38.65% | 86.79% | 36.82% | 0.430
Our Baseline | 512 | 67.27% | 98.54% | 68.21% | 0.717

###  4.6 Compare With State-of-the-art Models

We adopted some existing state-of-the-art methods for training and testing in
the DenseUAV dataset, and R@1, R@Top1%,mAP and our proposed SDM evaluation
metrics were applied to evaluate the performance. As shown in Table 6, our
Baseline improved nearly 29 points in R@1 compared to MSBA, nearly 12 points
in R@Top1%, nearly 31 points in mAP, and nearly 29 points in our SDM1 metric.
The experiments show that our baseline has a strong ability to identify
spatial location information.

Table 7: Effect of flight altitude on SDM indicators.

Height | SDM1 | SDM3 | SDM5 | SDM10
---|---|---|---|---
80m | 0.706 | 0.681 | 0.597 | 0.445
90m | 0.724 | 0.706 | 0.623 | 0.471
100m | 0.722 | 0.699 | 0.626 | 0.485

###  4.7 Robustness of flight altitude

Considering the height uncertainty of UAVs during actual flight, we designed
experiments to verify the robustness of our baseline to UAV image scale
variations. We divide the drone-view image sets of 80, 90, and 100m (relative
ground height) from the test set and verify the results on the satellite
gallery of DenseUAV. As shown in Table 7, the SDM metrics are close for
different altitudes. The experiments show that our model has excellent
robustness to the scale changes generated by the UAV flight altitude.

##  5 Application Analysis

Another contribution of this paper is to propose a UAV-based visual
localization scheme and an improved neighbor-based retrieval method. Besides,
this paper analyzes the reasons of why large distance deviations occur. In
this paper, we introduce the general UAV localization scheme in Section 5.1.
And analyze the causes of mislocation in Section 5.2. And introduce the
neighbor-serach UAV localization scheme in Section 5.3.

Figure 7: Comparison of Global Search and Neighbor-Search methods.

###  5.1 Basic Global Search Method

The whole task of cross-view geo-localization is understood as an end-to-end
system, with an input of a drone-view image and an output of the corresponding
GPS information. For Visualization, we intercept part of the satellite map. As
shown in Figure 7, the blur serial number represents the real GPS location of
the UAV, while the red serial number is the GPS location predicted by the end-
to-end system mentioned above, and the same number refers to a corresponding
relationship. The global search method is subject to a large offset of the
serial number 4 in Figure 7(a), for which we do a specific analysis in Section
5.2 for this kind of mislocation.

Most of the samples are predicted to be in the vicinity of the true position,
which is because the displacement of the UAV in the air corresponds to a small
change in its captured images. In addition, the satellite image library
retrieved as a gallery is also discrete segmented. Our satellite images are
sampled from google earth, and our sampling span is 1/4 of the satellite image
size, for example, if the size of satellite image that we cropped is
640$\times$640, then the span between adjacent satellite images is 160 pixels.
This is one of the most important reasons for the position shift in the
matched images.

Figure 8: Analysis diagram of the mislocation. The yellow circle is the
position of the previous match, the green dashed line is the centre of the
yellow circle with $M$ as the search radius arc, the red circle is the real
position of the current UAV, the blue circle is the position of the satellite
image matched out of the current UAV image, and the serial number represents
the ranking of the feature distance.

###  5.2 Error Localization Analysis

In the process of UAV localization, positioning deviations is caused by many
objective reasons, we have mainly listed the following three points. First,
there are many shadow areas in the satellite maps, which makes the fine-
grained information of the images lost and increases the difficulty of
geographic location matching. In addition, satellite images cannot be acquired
in real time, and only historical satellite images can be obtained. The
difference of time will lead to more or less changes between satellite maps
and actual drone-view images, such as newly planted trees, newly built houses,
and even changes in building color, which will lead to the non-correspondence
of the contents in the images. Second, satellite images from different
locations may also be very similar, and it can be understood to have a wrong
match when the images contain very similar contents. To identify the cause of
the mislocation we further studied point 4 in Figure 7(a), where a large
deviation appears. As shown in Figure 8, the red circles in the diagram refer
to the position of the UAV and the blue circles with serial numbers refer to
the position of the satellite images in ascending order of feature distance.
The features of the mislocation satellite images have some similar
characteristics to the query in the bottom left corner of the Figure 8. This
situation will worsen as the satellite database grows larger. In view of this,
we propose a localization strategy based on neighbor-search to solve the
localization problem for large distance deviations. Third, the different
flight heights of UAVs will lead to inconsistent image scales, for example,
UAVs flying at 100 meters will contain more but smaller scenes than those
flying at 80 meters, and this scale inconsistency will directly affect the
accuracy of matching under extreme conditions.

###  5.3 Neighborhood Search Solution

The global search method not only generates positioning errors over large
distances, but also increases the time consumption of post-processing. Relying
on the spatial continuity of drone flight, we designed a neighbor-search
method to improve this situation. The neighbor-search method is to narrow down
the original global domain to a neighborhood domain. As shown in Figure 8, the
yellow circle represents the location of the drone’s last match and the green
dotted line represents the area searched. It can be seen that the best match,
satellite map number 1, was excluded from the search, while satellite maps 11,
12, 13, 17 and 20, which were not ranked highly before, became the new best
choices. In terms of implementation, we reconstruct the candidate database by
filtering out satellite images whose spatial distance from the domain centroid
is less than a certain threshold on the basis of the global satellite
database. We then perform a search in this newly composed database for the
satellite image with the closest feature distance. To clearly demonstrate the
entire neighbor-search process, we visualise it in Figure 7(b). By comparing
the results of the matching in Figure 7(a) and (b), we can conclude that the
reduction of the retrieval domain will reduce the localization deviation.
Reasonable selection of the area to be retrieved not only reduces the
calculation time of the feature distance, but also improves the accuracy of
the UAV positioning. However, this method of neighbor-search also has some
limitations and will no longer be applicable when the spatial span of UAV
sampling is large.

##  6 Conclusion

This paper contributes a dense UAV-based geo-localization dataset named
DenseUAV to solve the problem of UAV self-localization. Based on this dataset,
a new evaluation metric for measuring spatial distance named SDM is proposed
to evaluate the performance of the algorithm in practical applications. In
addition, a Transformer-based baseline is proposed to implement end-to-end
training. In particular, a neiborgh-search post-processing strategy, which not
only reduces the time consumption of UAV localization but avoids large
position deviations, is proposed to optimize the UAV self-localization task.

Based on the idea of narrowing the retrieval range, there are many other
schemes worth trying. The retrieval range can be further reduced according to
the UAV’s IMU and flight speed, thus further improving the positioning
accuracy and reducing the positioning time consumption. In addition, the
navigation problem will be solved after achieveing the precise positioning of
UAVs.

## References

  *   * Arandjelovic et al (2016) Arandjelovic R, Gronat P, Torii A, et al (2016) Netvlad: Cnn architecture for weakly supervised place recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5297–5307
  * Carion et al (2020) Carion N, Massa F, Synnaeve G, et al (2020) End-to-end object detection with transformers. In: European Conference on Computer Vision, Springer, pp 213–229
  * Dosovitskiy et al (2020) Dosovitskiy A, Beyer L, Kolesnikov A, et al (2020) An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:201011929
  * He et al (2021) He S, Luo H, Wang P, et al (2021) Transreid: Transformer-based object re-identification. arXiv preprint arXiv:210204378
  * Hu and Lee (2020) Hu S, Lee GH (2020) Image-based geo-localization using satellite imagery. International Journal of Computer Vision 128(5):1205–1219. 10.1007/s11263-019-01186-0
  * Hu et al (2018) Hu S, Feng M, Nguyen RM, et al (2018) Cvm-net: Cross-view matching network for image-based ground-to-aerial geo-localization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 7258–7267
  * Jiang et al (2021) Jiang Y, Chang S, Wang Z (2021) Transgan: Two transformers can make one strong gan. arXiv preprint arXiv:210207074 1(3)
  * Lin et al (2015) Lin TY, Cui Y, Belongie S, et al (2015) Learning deep representations for ground-to-aerial geolocalization. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5007–5015
  * Liu et al (2017) Liu H, Feng J, Qi M, et al (2017) End-to-end comparative attention networks for person re-identification. IEEE Transactions on Image Processing 26(7):3492–3506
  * Liu and Li (2019a) Liu L, Li H (2019a) Lending orientation to neural networks for cross-view geo-localization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 5624–5633
  * Liu and Li (2019b) Liu L, Li H (2019b) Lending orientation to neural networks for cross-view geo-localization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 5624–5633
  * Liu et al (2021) Liu Z, Lin Y, Cao Y, et al (2021) Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:210314030
  * Long et al (2017) Long Y, Gong Y, Xiao Z, et al (2017) Accurate object localization in remote sensing images based on convolutional neural networks. IEEE Transactions on Geoscience and Remote Sensing 55(5):2486–2498. 10.1109/TGRS.2016.2645610
  * Ma et al (2021) Ma J, Jiang X, Fan A, et al (2021) Image matching from handcrafted to deep features: A survey. International Journal of Computer Vision 129(1):23–79. 10.1007/s11263-020-01359-2
  * McManus et al (2014) McManus C, Churchill W, Maddern W, et al (2014) Shady dealings: Robust, long-term visual localisation using illumination invariance. In: 2014 IEEE international conference on robotics and automation (ICRA), IEEE, pp 901–906
  * Mehta and Rastegari (2021) Mehta S, Rastegari M (2021) Mobilevit: Light-weight, general-purpose, and mobile-friendly vision transformer. arXiv preprint arXiv:211002178
  * Middelberg et al (2014) Middelberg S, Sattler T, Untzelmann O, et al (2014) Scalable 6-dof localization on mobile devices. In: European conference on computer vision, Springer, pp 268–283
  * Shi et al (2020a) Shi Y, Yu X, Campbell D, et al (2020a) Where am i looking at? joint location and orientation estimation by cross-view matching. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 4064–4072
  * Shi et al (2020b) Shi Y, Yu X, Liu L, et al (2020b) Optimal feature transport for cross-view image geo-localization. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp 11,990–11,997
  * Sun et al (2020) Sun Y, Cheng C, Zhang Y, et al (2020) Circle loss: A unified perspective of pair similarity optimization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 6398–6407
  * Tian et al (2017) Tian Y, Chen C, Shah M (2017) Cross-view image matching for geo-localization in urban environments. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 3608–3616
  * Toker et al (2021) Toker A, Zhou Q, Maximov M, et al (2021) Coming down to earth: Satellite-to-street view synthesis for geo-localization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 6488–6497
  * Touvron et al (2021) Touvron H, Cord M, Douze M, et al (2021) Training data-efficient image transformers & distillation through attention. In: International Conference on Machine Learning, PMLR, pp 10,347–10,357
  * Vaswani et al (2017) Vaswani A, Shazeer N, Parmar N, et al (2017) Attention is all you need. In: Advances in neural information processing systems, pp 5998–6008
  * Vo and Hays (2016) Vo NN, Hays J (2016) Localizing and orienting street views using overhead imagery. In: European conference on computer vision, Springer, pp 494–509
  * Wang et al (2021) Wang T, Zheng Z, Yan C, et al (2021) Each part matters: Local patterns facilitate cross-view geo-localization. IEEE Transactions on Circuits and Systems for Video Technology 10.1109/TCSVT.2021.3061265
  * Wang et al (2018) Wang X, Girshick R, Gupta A, et al (2018) Non-local neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7794–7803
  * Workman et al (2015) Workman S, Souvenir R, Jacobs N (2015) Wide-area image geolocalization with aerial reference imagery. In: Proceedings of the IEEE International Conference on Computer Vision, pp 3961–3969
  * Yang et al (2020) Yang F, Yang H, Fu J, et al (2020) Learning texture transformer network for image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 5791–5800
  * Yu et al (2020) Yu H, Li G, Zhang W, et al (2020) The unmanned aerial vehicle benchmark: Object detection, tracking and baseline. International Journal of Computer Vision 128(5):1141–1159. 10.1007/s11263-019-01266-1
  * Zhai et al (2017a) Zhai M, Bessinger Z, Workman S, et al (2017a) Predicting ground-level scene layout from aerial imagery. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 867–875
  * Zhai et al (2017b) Zhai M, Bessinger Z, Workman S, et al (2017b) Predicting ground-level scene layout from aerial imagery. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 867–875
  * Zhang et al (2020) Zhang H, Wu C, Zhang Z, et al (2020) Resnest: Split-attention networks. arXiv preprint arXiv:200408955
  * Zhao et al (2013) Zhao R, Ouyang W, Wang X (2013) Person re-identification by salience matching. In: Proceedings of the IEEE international conference on computer vision, pp 2528–2535
  * Zheng et al (2015) Zheng L, Shen L, Tian L, et al (2015) Scalable person re-identification: A benchmark. In: Proceedings of the IEEE international conference on computer vision, pp 1116–1124
  * Zheng et al (2021) Zheng S, Lu J, Zhao H, et al (2021) Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 6881–6890
  * Zheng et al (2020a) Zheng Z, Wei Y, Yang Y (2020a) University-1652: A multi-view multi-source benchmark for drone-based geo-localization. In: Proceedings of the 28th ACM international conference on Multimedia, pp 1395–1403
  * Zheng et al (2020b) Zheng Z, Zheng L, Garrett M, et al (2020b) Dual-path convolutional image-text embeddings with instance loss. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 16(2):1–23
  * Zhuang et al (2021) Zhuang J, Dai M, Chen X, et al (2021) A faster and more effective cross-view matching method of uav and satellite images for uav geolocalization. Remote Sensing 13(19):3979. 10.3390/rs13193979


# Generative and Teachers - For Us or Against Us? A Case Study

Jenny Pettersson, Elias Hult, Tim Eriksson and Tosin Adewumi1
Machine Learning Group, EISLAB, Luleå University of Technology, Sweden We
thank Björn Backe for his support for this work. This work was partially
supported by the Wallenberg AI, Autonomous Systems and Software Program
(WASP), funded by the Knut and Alice Wallenberg Foundation and counterpart
funding from Luleå University of Technology (LTU).1Corresponding author
<EMAIL_ADDRESS>
###### Abstract

We present insightful results of a survey on the adoption of generative
artificial intelligence (GenAI)by university teachers in their teaching
activities. The transformation of education by GenAI, particularly large
language models (LLM s), has been presenting both opportunities and
challenges, including cheating by students. We prepared the online survey
according to best practices and the questions were created by the authors, who
have pedagogy experience. The survey contained 12 questions and a pilot study
was first conducted. The survey was then sent to all teachers in multiple
departments across different campuses of the university of interest in Sweden:
Luleå University of Technology. The survey was available in both Swedish and
English. The results show that 35 teachers (more than half) use GenAI out of
67 respondents. Preparation is the teaching activity with the most frequency
that GenAI is used for and ChatGPT is the most commonly used GenAI. 59% say it
has impacted their teaching, however, 55% say there should be legislation
around the use of GenAI, especially as inaccuracies and cheating are the
biggest concerns.

##  I INTRODUCTION

Recent advances in artificial intelligence (AI), especially generative
artificial intelligence (GenAI), have caused a stir in the Education sector
around the world [1, 2]. ChatGPT111chat.openai.com/, the leading Large
Language Model (LLM)by OpenAI, has both been beneficial and controversial.
Some of the concerns with GenAI are generating deepfakes and the nature of
these "unexplainable" models [11]. In spite of these concerns, many recognize
the benefits inherent in these technologies [7, 11]. In this case study, we
seek to understand how university teachers perceive GenAI and investigate the
following research questions.

  1. 1.

To what extent are university teachers open to adopting generative AI in their
teaching and classrooms?

  2. 2.

What is the correlation between the impact of GenAI on teachers’ teaching
activities and their encouragement of their students to use it?

There’s increasing study of the impact of GenAI on students [7, 11]. It is
equally important to study the impact on other stakeholders or teachers’
teaching activities in Education [9]. United Nations Educational, Scientific
and Cultural Organization (UNESCO)hopes GenAI will be a tool that benefits
teachers, students, and researchers. Our main contributions include the
following: (1) We show through data that university teachers in this case
study are open to adopting GenAI; (2) we demonstrate the correlation between
the positive impact of GenAI on teachers to their willingness to encourage
their students to adopt it; (3) we provide many qualitative examples of
comments of teachers on the impact of GenAI, ways they encourage students, and
their concerns.

The rest of this paper is organised as follows. The literature review is
discussed in Section II. The method employed in this work is described in
detail in Section III. The findings are discussed in Section IV. We conclude
with closing remarks in Section V.

##  II Literature Review

The subject of GenAI in teaching is gaining increasing attention. It’s impact
on pedagogy cannot be ignored. Recent LLM s, such as ChatGPT, Aurora-M [15],
Large Language Model Meta AI (LLaMA)-2 [21] and a host of others, have
compelling abilities to generate human-like content, based on their training
with big data [11]. This has prompted United Nations Educational, Scientific
and Cultural Organization (UNESCO)to publish the guidance for GenAI in
education and research [11], which builds on their recommendation on the
ethics of AI. To gauge the awareness of teachers or educators and their
adoption of GenAI, it is useful to conduct a survey, similarly to that done
with students [6, 20].

[14] identified some of the best practices for designing the questionnaire for
a survey. As simple as it may sound, the ordering of questions is an important
consideration [12, 13]. These best practices are essential to have quality
data from the survey. [14] also show that it is important to design questions
to avoid acquiescence bias, which is the endorsement of a statement,
regardless of the content. Testing a survey in a pilot study usually improves
the quality of the full survey, as emphasized by [16, 17, 4, 5]. Using either
closed or open questions have their own benefits, as demonstrated by [19, 18,
8]. In our work, we combined both types of questions to get the best our of
the survey.

##  III Method

Our chosen method for evaluating the use of GenAI by teachers at LTU is
through a survey. It was designed to gather information about their habitual
use of GenAI in teaching. In creating the survey, a few factors were taken
into consideration to keep the inquiry objective and to avoid phrasings that
could potentially skew the answers of the participants, based on best
practices [14] . The factors are given below but are not limited to them.

  * •

Keep the questions simple and concise, so as not to produce off-topic answers
[10, 3].

  * •

Include a broad, exhaustive list of viewpoints [14]

  * •

Avoid questions where people tend to agree or disagree with statements
regardless of their actual feelings or beliefs [14]

  * •

Avoid single or double negation questions [14].

To reach a larger target audience, the survey questionnaire was created in
both Swedish and English using an online tool.222Miscrosoft Forms The Swedish
translation was carried out by some of the authors of this work, who are
native speakers. The translations in the Results and Appendix sections were
machine-translated and vetted by the native speakers. The survey consists of
12 questions to cover a relatively broad range of concepts that are necessary
to answer the research questions. For the purpose of the survey, a simplified
definition was provided for GenAI: Generative AI, such as ChatGPT or DALL-E,
is a tool that can answer questions and create images and other media based on
prompts from the user [11]. A copy of the questionnaire is available
online.333forms.office.com/e/jvrmPPaJJh

The survey was anonymous and a pilot study involving 8 teachers was initially
carried out to ascertain if the forms and questions needed any adjustments.
This full study involved sending the online survey via a link to all the
teachers in multiple departments in different geographical campuses of LTU.
The following are the 12 questions in the survey and their answer options, as
created by the authors based on their pedagogy experience and the factors
mentioned earlier. Q1 and Q8 were multiple choice questions. For teachers who
answer “None” to the first question ("filter question" [14]), they were
directed to continue to question 6 onwards.

  1. 1.

Have you used any generative AI in any of your teaching activities (e.g.
preparation, teaching, assessment, or none)?
i) Preparation  ii) Teaching  iii) Assessment  iv) Research444Undersökning in
Swedish, though initially translated as Undervisning. v) Administration  vi)
None  vii) Other

  2. 2.

Which ones? e.g GenAI, Ex. ChatGPT, DALL-E, Bing AI, Google Bard etc. Others

  3. 3.

How often do you use the one you use most? i) Once a month  ii) Once a week
iii) Twice or more a week  iv) Less than once a month

  4. 4.

Do you think the use has impacted your teaching? i) Yes  ii) No  iii) Not sure

  5. 5.

Briefly describe the impact on your teaching.

  6. 6.

Do you think AI will replace teachers in your subject if the trend of AI
development continues? i) Yes  ii) No  iii) Not sure

  7. 7.

Do you think there should be legislation around the use of generative AI? i)
Yes  ii) No  iii) Ambivalent

  8. 8.

What are some of your ethical concerns about generative AI? i) Gender bias
ii) Racial bias  iii) Inaccuracies  iv) Cheating  v) None  vi) Other concerns

  9. 9.

Will you encourage any of your students to use generative AI (in an ethical
manner)? i) Yes  ii) No

  10. 10.

If you answered “Yes” in the previous question, In what way? And if "No",
please say why.

  11. 11.

Your gender i) Woman  ii) Man  iii) Non-binary  iv) Prefer not to say

  12. 12.

Your Division and Department

The following LTU departments were involved in filling the survey.

  1. 1.

SRT: The Department of Computer Science, Electrical and Space Engineering,
among other subjects, contains Pervasive and Mobile Computing, Digital
Services and Systems, Computer Science, Signals and Systems, Robotics and AI,
Space Technology, Cyber-Physical Systems and Machine Learning.

  2. 2.

SBN: The Civil, Environmental and Natural Resources Engineering department,
among other subjects, contains Urban Water Engineering, Architecture,
Structural Engineering, Building Materials, Engineering Acoustics, Soil
Mechanics, Ore Geology, Applied Geophysics, Applied Geochemistry, Chemical
Technology, and Process Metallurgy

  3. 3.

ETKS: The Department of Social Sciences, Technology and Arts, among other
subjects, contains Industrial Marketing, Political Science, Human Work
Sciences, Performing Arts, Musical Performance, Economics, and Design.

  4. 4.

HLT: The Department of Health, Education and Technology, among other subjects,
contains Occupational Therapy, English and Education, Physiotherapy,
Biomedical Engineering, Medical Science, Nursing, and Psychology.

  5. 5.

TVM: The Department of Engineering Sciences and Mathematics, among other
education, contains Mechanical Engineering, Automotive Engineering,
Sustainable Energy, Electrical Power, Engineering Physics, and Mechanical
Engineering.

##  IV Results and Discussion

The survey took 3:53 minutes for each teacher to complete on average. From
Table I, 32 (48%) of the teachers do not use GenAI in any of their teaching
activities. The remaining 35 (52%) use GenAI for one or more teaching
activities, where Preparation is the most frequent activity, being 27%.
Besides the identified teaching activities, one teacher explained that she
uses GenAI "To see if student work is AI generated". The wordcloud of Figure 1
shows that ChatGPT has the lion share of usage with 52% of frequency of
mentions (26 out of 50), in answer to Q2. The nearest is DALL-E, with 6%.
Gemini, Stable Diffusion Web, and Midjourney come in next at 4% while all the
other GenAI s have 2% (only 1 mention). Figure 2 shows the distribution of
activities across gender.

TABLE I: Results in percentage (%). The gender and department sections each add up to the total. Question |  Option |  Total | Gender |  | Department |
---|---|---|---|---|---|---
|  |  |  M |  W |  Not say |  SRT |  SBN |  ETKS |  HLT |  TVM
Q1 |  Preparation |  27 |  15 |  10 |  2 |  3.6 |  3.6 |  14.4 |  3.6 |  1.8
Teaching |  14 |  10 |  2 |  2 |  6 |  2 |  6 |  0 |  0
Assessment |  3 |  3 |  0 |  0 |  0 |  0 |  0 |  0 |  3
|  Research |  20 |  9 |  10 |  1 |  7.5 |  5 |  7.5 |  0 |  0
|  Administration |  3 |  3 |  0 |  0 |  3 |  0 |  0 |  0 |  0
|  None |  32 |  17 |  13 |  2 |  9.41 |  7.53 |  9.41 |  5.65 |  0
|  Other |  1 |  0 |  1 |  0 |  0 |  1 |  0 |  0 |  0
Q3 |  Once a month |  18.42 |  13.16 |  5.26 |  0 |  3.07 |  6.14 |  6.14 |  3.07 |  0
Once a week |  18.42 |  10.53 |  7.89 |  0 |  9.21 |  0 |  9.21 |  0 |  0
> once a week |  31.58 |  15.79 |  7.89 |  7.89 |  21.05 |  0 |  5.26 |  0 |  5.26
|  < once a month |  31.58 |  15.79 |  15.79 |  0 |  3.95 |  11.84 |  11.84 |  3.95 |  0
Q4 |  Yes |  58.97 |  41.03 |  12.82 |  5.13 |  19.66 |  14.74 |  19.66 |  0 |  4.91
No |  17.95 |  10.26 |  7.69 |  0 |  0 |  8.97 |  4.49 |  4.49 |  0
Not sure |  23.08 |  5.13 |  15.38 |  2.56 |  11.54 |  0 |  7.69 |  3.85 |  0
Q6 |  Yes |  4.48 |  4.48 |  0 |  0 |  4.48 |  0 |  0 |  0 |  0
No |  85.07 |  41.79 |  35.82 |  7.46 |  29.10 |  15.67 |  29.10 |  11.19 |  0
Not sure |  10.45 |  8.96 |  1.49 |  0 |  3.48 |  3.48 |  1.74 |  0 |  1.74
Q7 |  Yes |  55.22 |  29.85 |  19.40 |  5.97 |  23.36 |  12.74 |  14.87 |  2.12 |  2.12
No |  17.91 |  11.94 |  4.48 |  1.49 |  5.12 |  5.12 |  7.67 |  0 |  0
Ambivalent |  26.87 |  13.43 |  13.43 |  0 |  6.72 |  2.24 |  6.72 |  11.19 |  0
Q8 |  Gender bias |  15.34 |  7.98 |  5.52 |  1.84 |  5.41 |  1.80 |  6.32 |  0.90 |  0.90
Racial bias |  15.34 |  7.36 |  6.13 |  1.84 |  5.96 |  1.70 |  5.96 |  0.85 |  0.85
Inaccuracies |  34.36 |  18.40 |  13.50 |  2.45 |  11.75 |  7.23 |  10.85 |  3.62 |  0.90
|  Cheating |  26.99 |  15.34 |  10.43 |  1.23 |  8.70 |  4.35 |  7.84 |  5.22 |  0.87
|  None |  1.23 |  1.23 |  0 |  0 |  1.23 |  0 |  0 |  0 |  0
|  Other |  6.75 |  2.45 |  3.68 |  0.61 |  1.12 |  2.25 |  3.37 |  0 |  0
Q9 |  Yes |  76.12 |  44.78 |  26.87 |  4.48 |  30.45 |  17.40 |  23.92 |  2.17 |  2.17
No |  23.88 |  10.45 |  10.45 |  2.99 |  4.34 |  2.17 |  6.51 |  10.85 |  0
Q11 |  Woman |  37.31 |  |  |  |  |  |  |  |
Man |  55.22 |  |  |  |  |  |  |  |
Non-binary |  0 |  |  |  |  |  |  |  |
|  Prefer not say |  7.46 |  |  |  |  |  |  |  |

Of the 52% who use these tools, there are as many teachers who use them twice
or more in a week (i.e. more than once) as there are those who use them less
than once a month (31.58%). About 59% say GenAI has impacted their teaching.
Indeed, we see a strong positive correlation between the Yes choices of the
departments to Q4 (Do you think the use has impacted your teaching?) and Q9
(Will you encourage any of your students to use generative AI?), based on the
Spearman’s correlation coefficient ($\rho$) +0.9474,555r = 1 and -1 are
perfect positive and negative correlations, respectively for p = 0.01438
(2-tailed). This implies the association between the two variables can be
considered statistically significant. Several examples of the ways teachers
feel GenAI has impacted their teaching are given in the appendix.

Figure 1: WordCloud of GenAI Figure 2: GenAI usage activities across gender
out of a total percentage of 100%. Figure 3: GenAI concerns across departments
out of a total percentage of 100%.

Most (85%) of the 67 teachers do not think AI will replace teachers. Many
(55%) say there should be legislation around the use of GenAI, possibly
because of the ethical concerns observed. Inaccuracies (34.36%) and cheating
(26.99%) are the two most common concerns teachers have. Figure 3 shows the
concerns across the departments. Given that the majority of teachers encourage
their students to use GenAI, some of the ways they go about it are listed in
the appendix, including these:

  * •

"…I do 1 or 2 sessions on how and why they should use it. I also show them
when it can give wrong results and how to fact check it…"

  * •

"seed texts, help in checking texts and results; use AI as study buddy"

  * •

"Det är ett mycket effektivt hjälpmedel och bör uppmuntras. Är man orolig för
fusk så examinerar man studenterna på fel sätt. (It is a very effective aid
and should be encouraged. If you are worried about cheating, you are examining
the students in the wrong way.)"

Overall, 37.31%, 55.22%, and 7.46% of the teachers who completed the survey
were women, men and those who preferred not to say.

##  V Conclusion

We have shown in this case study that teachers are open to adopting GenAI, as
over 50% currently use it. We also observe a strong positive correlation
between the positive impact of GenAI on their teaching activities and their
willingness to encourage their students to adopt GenAI. We agree with the
comments of some of the teachers that students "are guaranteed to use it
already", therefore we believe teachers should be knowledgeable about these
tools in order to provide the appropriate guidance for students. Future work
can investigate some of the ways of addressing the concerns of teachers
expressed in this study.

## ACKNOWLEDGMENT

We deeply appreciate all the teachers who completed the survey. Their valuable
response made this work possible. We thank Björn Backe for his support towards
this work. This work was partially supported by the Wallenberg AI, Autonomous
Systems and Software Program (WASP), funded by the Knut and Alice Wallenberg
Foundation and counterpart funding from Luleå University of Technology (LTU).

## APPENDIX

### Examples by teachers of impact on teaching

  1. 1.

I am able to provide more bang for the bucks and student feedback shows the
impact.

  2. 2.

I made my subject knowledge deeper.

  3. 3.

ChatGPT hjälper mig att sammanfatta innehållet i t ex en workshop eller en
föreläsning (ChatGPT helps me summarize the content of, for example, a
workshop or a lecture)

  4. 4.

t.ex. bättre bilder till mitt undervisningsmaterial, hjälp med bra
översättning till engelska etc. (for example better pictures for my teaching
material, help with a good translation into English, etc.)

  5. 5.

preparation of slides and text material is more efficient and result more
impactful.

  6. 6.

Bättre språk, fler exempel (Better language, more examples)

  7. 7.

I show its use for students to use properly as a tool, as well as when or how
not to use it.

  8. 8.

improved clarity

  9. 9.

Jag får ett bollplank som kan hjälpa mig som lärare. (I get a sounding board
that can help me as a teacher.)

  10. 10.

Jag har använt Canva:s generativ AI för att snabbt ta fram illustrationer till
mina powerpoints. Jag tror att det kan ha en viss positiv påverkan för
inlärning att få "bildstöd" till anteckningarna. (I have used Canva’s
generative AI to quickly produce illustrations for my powerpoints. I think
having "visual support" for the notes can have some positive impact on
learning.)

  11. 11.

Tidsbesparande (Timesaving)

  12. 12.

Ibland dyker det upp aspekter som jag tidigare inte tänkt på, men som är
relevanta. (Sometimes aspects appear that I previously did not think about,
but which are relevant.)

  13. 13.

Är ett fantastiskt verktyg att skapa bilder istället för att leta clip-art.
Att få hjälp att förklara saker samt som kreativt verktyg i
idegenereringsprocessen (Is a great tool to create images instead of looking
for clip-art. To get help explaining things and as a creative tool in the idea
generation process)

  14. 14.

Jag har fått en bättre förståelse för hur studenter kan använda det som stöd
samt jag har lärt mig att känna igen resultaten i studenters arbete (I have
gained a better understanding of how students can use it as support and I have
learned to recognize the results of students’ work)

### Examples of other concerns

  1. 1.

I have had trouble with students copying AI generated information. I am afraid
they are not using it as a learning tool, but rather to avoid learning.

  2. 2.

…students’ usage affects the type of tasks I can give them and how I test
their knowledge. That’s mainly why I use it myself.

  3. 3.

Nya former för examination kräva (New forms of examination require)

  4. 4.

Efter att tidigare ha använt hemtentor i delar av kursen har jag gått över
till salskrivningar. (Having previously used take-home exams in parts of the
course, I have switched to classroom writing.)

  5. 5.

Jag har tydligt kunnat visa för mina studenter varför det är viktigt att kunna
ha grundläggande kunskap inom ett område, för det AI säger behöver inte
nödvändigtvis vara korrekt, vilket de fick erfara i en kurs. (I have been able
to clearly show my students why it is important to be able to have basic
knowledge in a field, because what the AI says does not necessarily have to be
correct, as they experienced in a course.)

  6. 6.

I sin nuvarande form är generativ AI bra på att generera text som ser rimlig
ut men mycket väl kan vara ful av felaktigheter. Jag ser inte detta som
särskilt anändbart för mina studenter. (In its current form, generative AI is
good at generating text that looks reasonable but may well be ugly with
inaccuracies. I don’t see this as particularly relevant to my students.)

### Examples of ways teachers encourage their students to use GenAI

  1. 1.

"Use it to learn, not to cheat". Använd för att förbättra eget material, inte
för att generera från grunden. Viktigt att man inte presenterar andras
material som sitt eget. Däremot är det liten skillnad att få en
språkfgransking av en människa eller från AI när man väl skrivit texten.
Viktigt att kunna materialet så att man kan faktagranska AI-lösningarna. ("Use
it to learn, not to cheat". Use to improve your own material, not to generate
from scratch. It is important not to present other people’s material as your
own. However, there is little difference in getting a language check by a
human or from AI once you have written the text. It is important to know the
material so that you can fact-check the AI solutions.)

  2. 2.

Jag uppmuntrar dem att använda AI så mycket som möjligt om det hjälper deras
lärande. (I encourage them to use AI as much as possible if it helps their
learning.)

  3. 3.

T.ex. för att bolla idéer, få hjälp med struktur i en text, hitta och sortera
källor. Jag tycker det är ett väldig kraftfylld verktyg men precis som vilken
verktyg som helst kan den vara farlig om den används av människor utan rätt
kunskap. Så att skaffa sig just den kunskapen för att kunna använda generativ
AI på ett säkert sätt bor ingår undervisningen. (For example. to brainstorm
ideas, get help with structure in a text, find and sort sources. I think it’s
a very powerful tool but like any tool it can be dangerous if used by people
without the right knowledge. So acquiring that particular knowledge to be able
to use generative AI in a safe way is part of the teaching.)

  4. 4.

De använder det garanterat redan, så bättre att ha riktlinjer kring hur
användandet bör ske. (They are guaranteed to use it already, so better to have
guidelines about how the use should take place.)

  5. 5.

Inte uppmuntra, men inte heller hindra (Not encouraging, but not hindering
either)

  6. 6.

Framförallt för att lära sig skriva vetenskaplig text på engelska och för att
diskutera kursinnehåll, mjukvarukunskaper (t.ex. Excel, Matlab, python etc.)
(Mainly to learn how to write scientific text in English and to discuss course
content, software skills (e.g. Excel, Matlab, python etc.))

  7. 7.

Studenterna (Och industrin) använder redan generativ AI i långt högre
utsträckning än vad vi lärare gör. Bättre att lära från dem och uppmana dem
till att använda systemen på ett klokt sätt som uppmuntrar deras lärande.
(Students (And industry) already use generative AI to a far greater extent
than we teachers do. Better to learn from them and encourage them to use the
systems wisely which encourages their learning.)

  8. 8.

Precis som jag skrev innan så har jag uppmuntrat studenterna att använda
ChatGPT för att få förståelse/fördjupning av vissa ämnen (Just as I wrote
before, I have encouraged the students to use ChatGPT to gain
understanding/deepening of certain topics)

  9. 9.

Leta material, sortera i material (Find materials, sort in materials)

  10. 10.

ja, att använda det i den kreativa processen för att utforska en mängd idéer,
för att förbättra engelskan i texter etc. (yes, to use it in the creative
process to explore a variety of ideas, to improve English in texts, etc.)

  11. 11.

Som ett skrivstöd (As a writing aid)

  12. 12.

För språkgranskning och programmering (For language review and programming)

  13. 13.

AI kan vara mycket hjälpfullt om man har ett koncept/begrepp som man inte
förstår men vill ha förklarat så att man sen, självständigt, kan använda
konceptet/begreppet. (AI can be very helpful if you have a concept that you
don’t understand but want explained so that you can then, independently, use
the concept.)

  14. 14.

Utmärkt för sammanfattningar och scanning av stora litteraturmängder
(Excellent for summaries and scanning large volumes of literature)

  15. 15.

utkast till texter, utkast till musik, tex. (drafts of texts, drafts of music,
e.g.)

  16. 16.

Det är ett ypperligt bollplank, framförallt då man inte har någon fysisk
person att diskutera med, men det kan även ge förstärkning om man använder den
då man är studerar i grupp. (It is an excellent sounding board, especially
when you have no physical person to discuss with, but it can also provide
reinforcement if you use it when you are studying in a group.)

  17. 17.

I produktutveckling, som ett stöd och verktyg. (In product development, as a
support and tool.)

  18. 18.

skriv hjälp, hjälp att komma igång med ett arbete, skapa bilder för
presentationer, hjälp mot skrivkramp (write help, help to get started with a
work, create images for presentations, help against writing cramp)

  19. 19.

1\. Använda det för att stava rätt i inlämningar. 2. För att ha någon att
bolla idéer kring studentarbete. 3. Fråga om grundläggande koncept som AI kan
behärska och förklara. (1\. Use it to spell correctly in submissions. 2. To
have someone to bounce ideas off of student work. 3. Ask about basic concepts
that AI can master and explain.)

  20. 20.

Vi lärare behöver skapa och formulera premisser som är rimliga för detta, nu
när man inte kan backa bandet med AI. Hur - vet jag inte än. (We teachers need
to create and formulate premises that are reasonable for this, now that you
cannot reverse the trend with AI. How - I don’t know yet.)

## References

  * [1] Adewumi, T., Alkhaled, L., Buck, C., Hernandez, S., Brilioth, S., Kekung, M., Ragimov, Y., and Barney, E. Procot: Stimulating critical thinking and writing of students through engagement with large language models (llms).  arXiv preprint arXiv:2312.09801 (2023).
  * [2] Alasadi, E. A., and Baiz, C. R. Generative ai in education and research: Opportunities, concerns, and solutions.  Journal of Chemical Education 100, 8 (2023), 2965–2971.
  * [3] Artino Jr, A. R., and Gehlbach, H. Am last page: Avoiding four visual-design pitfalls in survey development.  Academic Medicine 87, 10 (2012), 1452.
  * [4] Baburajan, V., e Silva, J. d. A., and Pereira, F. C. Open-ended versus closed-ended responses: A comparison study using topic modeling and factor analysis.  IEEE transactions on intelligent transportation systems 22, 4 (2020), 2123–2132.
  * [5] Bowden, A., Fox-Rushby, J., Nyandieka, L., and Wanjau, J. Methods for pre-testing and piloting survey questions: illustrations from the kenqol survey of health-related quality of life.  Health policy and planning 17, 3 (2002), 322–330.
  * [6] Chan, C. K. Y., and Hu, W. Students’ voices on generative ai: Perceptions, benefits, and challenges in higher education.  International Journal of Educational Technology in Higher Education 20, 1 (2023), 43.
  * [7] Chan, C. K. Y., and Lee, K. K. The ai generation gap: Are gen z students more interested in adopting generative ai such as chatgpt in teaching and learning than their gen x and millennial generation teachers?  Smart Learning Environments 10, 1 (2023), 60.
  * [8] Connor Desai, S., and Reimers, S. Comparing the use of open and closed questions for web-based measures of the continued-influence effect.  Behavior research methods 51 (2019), 1426–1440.
  * [9] Dai, Y., Liu, A., and Lim, C. P. Reconceptualizing chatgpt and generative ai as a student-driven innovation in higher education.  Procedia CIRP 119 (2023), 84–90.
  * [10] Gehlbach, H., and Artino Jr, A. R. The survey checklist (manifesto).  Academic Medicine 93, 3 (2018), 360–366.
  * [11] Holmes, W., Miao, F., et al. Guidance for generative AI in education and research.  UNESCO Publishing, 2023.
  * [12] Jensen, P. S., Watanabe, H. K., and Richters, J. E. Who’s up first? testing for order effects in structured interviews using a counterbalanced experimental design.  Journal of Abnormal Child Psychology 27 (1999), 439–445.
  * [13] Kreuter, F., McCulloch, S., Presser, S., and Tourangeau, R. The effects of asking filter questions in interleafed versus grouped format.  Sociological Methods & Research 40, 1 (2011), 88–104.
  * [14] Krosnick, J. A. Questionnaire design.  The Palgrave handbook of survey research (2018), 439–455.
  * [15] Nakamura, T., Mishra, M., Tedeschi, S., Chai, Y., Stillerman, J. T., Friedrich, F., Yadav, P., Laud, T., Chien, V. M., Zhuo, T. Y., Misra, D., Bogin, B., Vu, X.-S., Karpinska, M., Dantuluri, A., jciech Kusa, W., Furlanello, T., Yokota, R., Muennighoff, N., Pai, S., Adewumi, T. P., Laippala, V., Yao, X., Junior, A., Ariyak, A., Drozd, A., Clive, J., Gupta, K., Chen, L., Sun, Q., Tsui, K., Persaud, N., Fahmy, N., Chen, T., Bansal, M., Monti, N., Dang, T., Luo, Z., Bui, T.-T., Navigli, R., Mehta, V., Blumberg, M., May, V., Nguyen, H., and Pyysalo, S. Aurora-m: The first open source multilingual language model red-teamed according to the u.s. executive order.
  * [16] Oksenberg, L., and Kalton, G. New strategies for pretesting survey questions.  Journal of official statistics 7, 3 (1991), 349.
  * [17] Presser, S., Couper, M. P., Lessler, J. T., Martin, E., Martin, J., Rothgeb, J. M., and Singer, E. Methods for testing and evaluating survey questions.  Methods for testing and evaluating survey questionnaires (2004), 1–22.
  * [18] Reja, U., Manfreda, K. L., Hlebec, V., and Vehovar, V. Open-ended vs. close-ended questions in web questionnaires.  Developments in applied statistics 19, 1 (2003), 159–177.
  * [19] Schuman, H., and Scott, J. Problems in the use of survey questions to measure public opinion.  Science 236, 4804 (1987), 957–959.
  * [20] Smolansky, A., Cram, A., Raduescu, C., Zeivots, S., Huber, E., and Kizilcec, R. F. Educator and student perspectives on the impact of generative ai on assessments in higher education.  In Proceedings of the tenth ACM conference on Learning@ Scale (2023), pp. 378–382.
  * [21] Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models.  arXiv preprint arXiv:2307.09288 (2023).