# Neighbor Oblivious Learning (NObLe) for Device Localization and Tracking

Zichang Liu  Department of Computer Science
Rice University
Houston, USA
<EMAIL_ADDRESS>Li Chou  Department of Computer Science
Rice University
Houston, USA
<EMAIL_ADDRESS>Anshumali Shrivastava  Department of Computer Science
Rice University
Houston, USA
<EMAIL_ADDRESS>
###### Abstract

On-device localization and tracking are increasingly crucial for various
applications. Along with a rapidly growing amount of location data, machine
learning (ML) techniques are becoming widely adopted. A key reason is that ML
inference is significantly more energy-efficient than GPS query at comparable
accuracy, and GPS signals can become extremely unreliable for specific
scenarios. To this end, several techniques such as deep neural networks have
been proposed. However, during training, almost none of them incorporate the
known structural information such as floor plan, which can be especially
useful in indoor or other structured environments. In this paper, we argue
that the state-of-the-art-systems are significantly worse in terms of accuracy
because they are incapable of utilizing this essential structural information.
The problem is incredibly hard because the structural properties are not
explicitly available, making most structural learning approaches inapplicable.
Given that both input and output space potentially contain rich structures, we
study our method through the intuitions from manifold-projection. Whereas
existing manifold based learning methods actively utilized neighborhood
information, such as Euclidean distances, our approach performs Neighbor
Oblivious Learning (NObLe). We demonstrate our approach’s effectiveness on two
orthogonal applications, including Wi-Fi-based fingerprint localization and
inertial measurement unit(IMU) based device tracking, and show that it gives
significant improvement over state-of-art prediction accuracy.

##  I Introduction

The global market size for location-based services is expected to grow to USD
26.7 billion by 2025 from USD 13.8 billion in 2020 [1]. The key to the
projected growth is an essential need for accurate location information. For
example, location intelligence is critical during public health emergencies,
such as the current COVID-19 pandemic, where governments need to identify
infection sources and spread patterns. Traditional localization systems rely
on global positioning system (GPS) signals as their source of information.
However, GPS can be inaccurate in indoor environments and among skyscrapers
because of signal degradation. Moreover, GPS is notorious for battery drainage
because of slow and demanding communication requirements [2]. Therefore, GPS
alternatives with higher precision and lower energy consumption are urged by
industry. Existing network infrastructure such as Wi-Fi (IEEE 802.11) is
utilized for localization [3][4] to avoid expensive infrastructure deployment.
Besides, low-cost inertial measurement sensors (IMU) based on accelerators and
gyroscopes, which are widely embedded in modern mobile devices, have also
emerged as popular solution [5][6] for both indoor and outdoor device tracking
task. An informative and robust estimation of position based on these noisy
inputs would further minimize localization error.

Machine learning (ML) techniques are a logical choice for these estimation
tasks, and popular algorithms such as $k$-nearest neighbors and random forest
have been proposed [7][8]. Since deep neural networks (DNN) have performed
surprisingly well in computer vision, natural language processing, and
information retrieval, many attempts have been made to utilize DNNs for
localization [9][10][11]. These approaches either formulate localization
optimization as minimizing distance errors or use deep learning as denoising
techniques for more robust signal features.

Figure 1: Both figures corresponds to the three building in UJIIndoorLoc
dataset. Left figure is the screenshot of aerial satellite view of the
buildings (source: Google Map). Right figure shows the ground truth
coordinates from offline collected data.

All the methods mentioned above fail to utilize common knowledge: space is
usually highly structured. Modern city planning defined all roads and blocks
based on specific rules, and human motions usually follow these structures.
Indoor space is structured by its design floor plan, and a significant portion
of indoor space is not accessible. As an illustration, consider Fig. 1 based
on the largest publicly available indoor Wi-Fi localization dataset
UJIIndoorLoc [12], which covers three buildings with four floors, a space of
397 meters by 273 meters. Space structure is clear from the satellite view,
and offline signal collecting locations exhibit the same structure. Fig. 4(a)
shows the outputs of a DNN that is trained using mean squared error to map Wi-
Fi signals to location coordinates. This regression model can predict
locations outside of buildings, which is not surprising as it is entirely
ignorant of the output space structure. It was observed in [8] [19] that
projecting the predicted outputs to the closest positions on the map would
increase localization precision. Our experiment shows that forcing the
prediction to lie on the map only gives marginal improvements. In contrast,
Fig. 4(d) shows the output of our NObLe model, and it is clear that its
outputs have a sharper resemblance to the building structures.

We view localization space as a manifold and our problem can be regarded as
the task of learning a regression model in which the input and output lie on
an unknown manifold. The high-level idea behind manifold learning is to learn
an embedding, of either an input or output space, where the distance between
learned embedding is an approximation to the manifold structure. In scenarios
when we do not have explicit (or it is prohibitively expensive to compute)
manifold distances, different learning approaches use nearest neighbors search
over the data samples, based on the Euclidean distance, as a proxy for
measuring the closeness among points on the actual manifold. While this is
justified because the definition of manifold states that any manifold locally
is a Euclidean space, however, the Euclidean distances between data pairs may
not be a good criterion for finding neighbors on manifold structures for
localization services as input signals are extremely noisy.

Our Contributions: Our proposal is inspired by approaches in manifold
learning. We argue that Euclidean distance is not reliable for local manifold
structure approximation in localization, and propose to ignore small changes
in the Euclidean distance and focus on the relative closeness of reconstructed
embedding. We propose Neighbor Oblivious Learning (NObLe), a DNN approach that
achieves structure-aware localization. Further, we demonstrate the
applicability of our techniques on two independent applications: (1) Wi-Fi
signal strength based indoor localization and (2) IMU-based device positioning
in an outdoor environment. Our evaluations on both applications show that
NObLe gives significant accuracy improvements. To illustrate that our system
can be deployed on energy and computation constraints mobile devices, we
thoroughly ran energy tests on two systems. We demonstrate that our model has
significantly smaller energy consumption (specifically, 27 times less energy
on IMU tracking) than GPS measurements.

##  II Background and Related Work

Manifold Learning: Manifold learning is a class of non-linear dimensionality
reduction methods. The objective is to find a low-dimensional representation
describing some given high-dimensional data observed from an input or feature
space $\mathcal{X}$. It is generally assumed that $\forall x\in\mathcal{X}$,
$x$ is sampled from some smooth $p$-dimensional submanifold
$\mathcal{M}\subset\mathbb{R}^{d}$. The manifold learning task is to then find
a mapping $\psi:x\rightarrow z\in\mathbb{R}^{s}$ such that $p\leq s\ll d$,
while, loosely stated, preserving (structural) properties (e.g., interpoint
distances) of the original feature space. Two popular manifold learning
methods are locally linear embedding (LLE) [13] and isometric mapping (Isomap)
[14]. These algorithms follow a template comprised of three steps: (1)
construct a neighborhood graph, which involves (expensive) nearest neighbor
search; (2) construct a (positive semi-definite) kernel, which is specified as
shortest path distances for Isomap, and weights (or coefficients) from solving
a system of linear equations for LLE; and (3) perform partial Eigenvalue
decomposition.

Wi-Fi Localization: It is cost-effective to leverage existing wireless
infrastructure to develop localization techniques. Combining Wi-Fi with radio
map is also known as fingerprinting, which consists of two phases. Offline
phase: signal features are sampled at selected locations and processed to
build the radio map, a database of locations, and their corresponding signal
values. One type of signal feature used is received signal strength indicator
(RSSI) values from multiple wireless access points (WAP). Online phase:
observed RSSI values are matched with points on the radio map to determine the
current location, which relies on searching for the most similar locations
based on the stored RSSI values in the radio map. Many of these techniques do
not use structural information.

Localization on IMU: Cheap inertial-based sensors on mobile computing devices
have emerged as a potential solution for infrastructure-free indoor
localization and navigation. However, there are two main challenges. First,
IMUs are extremely noisy, making it impossible to use only through physical
principles and numerical integration. Second, it keeps updating previous
positions, which makes it subject to error accumulation. Various techniques
have been proposed to mitigate error accumulation by ruling out illegal
movements. A line of work utilizes a floor map to hand-design heuristic rules
to correct localization error. For example, [8] achieved a mean error of 4.3m
on a testbed of 163m by 62m. With a map, it uses high-accuracy turn detection
to correct positioning error based on the assumption that turns can only be
made on specific points on the map.

ML in Localization: Several ML algorithms, such as support vector machines and
neural networks, have been applied to localization. Typically, signal strength
readings are used as inputs, and outputs are either two or three dimension
vectors, corresponding to 2-D or 3-D location estimates [22]. This approach
formulates localization as a regression problem that predicts two continuously
coordinate variable values given signal strength vector. ML is also used for
denoising in order to extract core features for wireless signals. WiDeep [10]
utilize one auto-encoder (AE) for every WAP, making it hard to scale. DeepFi
[9] also utilizes DNNs, but also ignore structure information. CNNLoc [11]
utilizes a complex architecture including stacked AEs and convolutional neural
networks to achieve a mean error of 11.78m on UJIIndoorLoc. ML was also
applied to IMU-based localization. [8] used nearest neighbors and random
forest regression to predict the travel distance based on IMU readings.

##  III Proposed System Design

###  III-A Intuition

The world we live in contains many structural themes and elements. Factoring
in structure information usually lead to performance improvement. For example,
in computer vision, many state-of-art approaches exploit structure within
images. Given the structural nature of localization space, we approach the
problem with the intuition and consideration that the input and output space
lies in a manifold space.

Manifold-based learning algorithms, usually unsupervised, utilize local
Euclidean distances to approximate neighborhood structure. However, the input
features for localization problems are noisy signals. When a person is
walking, the accelerometer and gyroscope sensors are likely to pick up a lot
of noise due to spurious movements. Moreover, different individuals have
different walking styles. Similarly, Wi-Fi signals can be noisy because of
moving crowds or room set-ups. Thus, small changes in such noisy input signals
are not reliable information about the manifold structure and direct adopting
traditional manifold learning approaches is not appropriate. To combat this
noise, we ignore small Euclidean differences and propose Neighbor Oblivious
Learning (NObLe). We propose to quantize the continuous output space into a
set of grid-like neighborhood areas, and all data points within the same grid
are considered belonging to the same class. It is widely accepted that the
penultimate layer of deep neural network classifier model can be regarded as
learned embedding [15] [16]. We use DNN and optimize it with cross entropy
loss to maximize the embedding distance between different classes, while
oblivious to embedding distance within the same class.

###  III-B Space Quantization and Multi-label Classification

Consider a space $S$ for localization. We collect data samples of the form
$(\vec{s},(x,y))$, where $\vec{s}$ is a vector representing signal features,
and $(x,y)$ denotes longitude and latitude coordinates. We propose to perform
space quantization on $(x,y)$ to transform continuous position coordinates
into neighborhood area classes. Each data sample now becomes
$(\vec{s},c,(x,y))$, where $c$ is a neighborhood area classes ID.
Specifically, we divide $S$ into non-overlapping square grids with a side
length of $\tau$. In practice, we set $\tau$ to be less than 0.2m. Then, we
assign each grid neighborhoods a class ID $c$ and discard all classes without
any data points. Thus, instead of using position coordinates as training
labels, NObLe uses neighborhood class as ground truth. During inference, NObLe
uses the predicted class to look up its neighborhood class’s central
coordinates and returns it as the prediction result.

Figure 2: Consider this space for localization. Gray circles represent
training data point locations. The whole space is quantized into four class
and data points are labeled with its corresponding grid class ID.

Our quantization approach exploits and approximates the ground truth closeness
between data points in the output space without relying on Euclidean distance
in the input space as neighborhood approximation. Moreover, assume a thorough
training data sampling process over space $S$, our method eliminates
inaccessible areas such as dead or irrelevant space from the output space
because samples either cannot be or are not intentionally collected from those
areas. For example, in Fig. 1, the middle area of top left buildings will not
translate to any neighborhood classes as no data resides in that area.

Our space quantization enables us to solve the manifold regression problem
with a fine-grained classification model. However, we have introduced one
hurdle. The classification problem is likely to suffer from class data
sparsity. Since our grid is fine-grained, it is likely to contain very few
training samples. We could increase $\tau$, or we could assign data samples
with multiple classes, the ones that are adjacent to the real class. Moreover,
we could also divide space $S$ into grid neighborhood of different length,
$\tau$ and $l$ where $\tau<l$ . Each data sample now becomes
$(\vec{s},c,r(x,y))$ where $c$ denotes for the neighborhood classes ID of size
$\tau$ and $r$ denotes neighborhood classes ID of size $l$. This formulation
gives different levels of granularity of the output manifold.

###  III-C Why DNN Classification is Equivalent to Manifold Learning

We will make the connection between manifold learning and our approach
mathematically. To begin with, we introduce multidimensional scaling (MDS), a
popular manifold learning algorithm, which has the objective:
$f(Z,\mathcal{X})=\sum_{i=1}^{n}\sum_{j=1+i}^{n}(||z_{i}-z_{j}||-||x_{i}-x_{j}||)^{2}$
for $n$ points. Essentially, MDS tries to learn embedding $Z$ on output
manifold such that the pairwise relationships on input space are preserved.
Close neighbors are encouraged to stay close in the reconstructed space and
vise versa.

In our formulation of NObLe, we use binary cross-entropy loss function for
multi-label classification, defined as
$J(h_{c},\hat{h}_{c})=\sum_{i=1}^{n}\sum_{c=1}^{k}-h_{c}\log(\hat{h}_{c})-(1-h_{c})\log(1-\hat{h}_{c})$,
where $k$ is the number of classes, $n$ is the number of training data,
$h_{c}\in\\{0,1\\}$ indicates the right class when $h_{c}=1$, and
$\hat{h}_{c}$ is the sigmoid function:
$\hat{h}_{c}=(1+\exp(-w_{c}^{\top}z_{i}))^{-1}$. Here, $w_{c}$ denotes the
weight vector for class $c$ at the last layer, and $z_{i}$ denotes for the
output from the second last layer for input $x_{i}$. We focus our analysis on
the last layer because the second last layer output can be interpreted as
learned embedding for input features. From a manifold learning perspective,
embedding from the last layer can be interpreted as reconstructed embedding.
For simplicity, suppose $w$ and $z$ are normalized, we can rewrite $\hat{h}$
from an inner product to the Euclidean distance form as
$\hat{h}_{c}=(1+\exp(\frac{1}{2}||w_{c}-z_{i}||^{2}-1))^{-1}.$

For a given $c$, minimizing the cross entropy loss will result in a setting
such that $||w_{c}-z_{i}||$ for the true class is minimized (cf. false class
is maximized). Consider $z_{i}$ as embedding given input $x_{i}$, $z_{j}$ as
embedding given input $x_{j}$. If $x_{i}$, $x_{j}$ are near neighbors, then by
our formulation, $x_{i}$, $x_{j}$ share same class label. Thus, the following
holds for two embedding $z_{i}$ and $z_{j}$, $||w_{c}-z_{i}||^{2}\leq\lambda$
and $||w_{c}-z_{j}||^{2}\leq\lambda$, where $\lambda$ is a small constant. And
by triangle inequality, we have $||z_{i}-z_{j}||^{2}\leq 2\lambda$. As we can
see, $z_{i}$, $z_{j}$ is expected to be close, which resembles the objective
function of MDS without considering the distance in the input space between
$x_{i}$ and $x_{j}$.

We present NObLe as a DNN based approach for localization that can utilize
structure information. It should be noted that our evaluation measure is still
position error (root mean square error) even though we transform the data into
fine-grained classification inspired by manifold learning. In the next two
sections, we will use NObLe on two orthogonal input signals, Wi-Fi for the
positioning task, and IMU for the tracking task.

##  IV Application Wi-Fi Localization

In this section, we first present the detailed system design of NObLe for Wi-
Fi fingerprinting localization. We conduct experiments on two representative
indoor Wi-Fi localization datasets: UJIIndoorLoc [12], the largest open-access
dataset for indoor Wi-Fi localization for large space multi-building setting,
and IPIN2016 [17] for small single building setting.

###  IV-A System Architecture

We follow the standard setup for Wi-Fi fingerprint localization. Assume there
are $W$ number of WAPs in the given space. During the offline phase, Wi-Fi
strength signal readings received from each WAP at each sampling location are
recorded. Floor, building, longitude, and latitude are also recorded for each
sample. Both UJIIndoorLoc and IPIN2016 are collected in such a way, and each
sample can be represented as $(\vec{s},b,f,(x,y))$.
$\vec{s}=(s_{1},s_{2},\ldots,s_{W})$, where $s_{i}$ denotes the RSS of $i$-th
WAP, $b$ denotes building ID, $f$ denotes floor ID. Given the collected data,
we perform output space quantization and convert each sample as
$(\vec{s},b,f,c,r,(x,y))$. Apply the NObLe multi-label classification
formulation, our model takes $\vec{s}$ as inputs, and predict $(b,f,c,r)$.
During inference, we use $c$ to look up the corresponding central coordinates,
and output $(x_{c},y_{c})$ as position and calculate position error
accordingly. One advantage of NObLe is that we can naturally include
floor/building classification tasks in our model without extra effort.
Floor/building classification is a standard task for localization service.
Current approaches utilize separate and independent models for position
prediction and building/floor classification, creating extra overhead in real-
world deployments. At the same time, from a manifold perspective, including
floor/building as output is beneficial for the model to learn the
reconstructed embedding because it gives useful information about geodesic
neighborhood over the manifold structure.

Figure 3: Given wireless signal strength as input, NObLe predicts multiple
labels, which includes C for neighborhood class, B for building, F for floor.
At inference time, NObLe computes longitude and latitude coordinates based on
the predicted neighborhood class.

We consider a two hidden layer feed-forward neural network that takes input
vector $\vec{s}\in\mathbb{R}^{W}$. The hidden layer size is set to 128. We
normalize the input vector and apply multi-hot encoding to the output class.
We used hyperbolic tangent activation functions, Xavier initialization [20],
and batch normalization [21] for training our model. The overview of our
system is shown in Fig. 3.

(a) Deep Regression

(b) Deep Regression Projection

(c) Isomap Regression ($d$=400)

(d) NObLe

Figure 4: Plots of predicted coordinates from four models (labeled below each
plot).

###  IV-B Performance Evaluation

In our experiments, we first show that NObLe achieves the best performance
compared to all other approaches on the same datasets. Moreover, we set up
three comparative models to demonstrate that NObLe is aware of the output
structure. We applied the best effort hyperparameter tuning for all methods.

TABLE I: NObLe performance results on UJIIndoorLoc. | Classification Accuracy (%)
---|---
Building | 99.74
Floor | 94.25
Quantize Class | 61.63
| Position Error Distances (m)
Mean | 4.45
Median | 0.23

We calculate position error following the standard procedure: the Euclidean
distance between predicted and true coordinates. For the UJIIndoorLoc dataset,
the best mean error distance on the indoor localization ranking at
IndoorLocPlatform website [17] is 6.2 m, and the median is 4.63m. [11] reports
a mean position error of 11.78 m, a building hit rate around 99%, and a floor
hit rate around 94%. [18] reports a mean position error of 9.29m, a building
hit rate around 99%, and a floor hit rate around 91%. As we can see in Table
I, NObLe achieves significantly smaller position error distances and at least
comparable building and floor hit rate.

In order to evaluate the performance improvement from the perspective of
structure awareness, we implement three comparison models: Deep Regression,
Deep Regression Projection, and Manifold Embedding. Deep Regression takes the
same input as NObLe. It is the same network size as NObLe. However, it is
trained with mean square error as loss function and directly predicts
coordinates in longitude and latitude. Deep Regression Projection is based on
[8]. Following Deep Regression, Deep Regression Projection projects the
predicted coordinates to the nearest position on the map when the predictions
do not lie on the map. Manifold Embedding utilizes Isomap and LLE to compute
embedding from input signals. We built DNNs with two hidden layers that take
the manifold embedding as input and output longitude and latitude coordinates.
Manifold Embedding achieves the best performance when we set the embedding
dimension at 400 for both Isomap and LLE. The performance results for the
models mentioned above are shown in Table II.

TABLE II: Comparative distance (m) errors on UJIIndoorLoc. Model | Mean | Median
---|---|---
Deep Regression | 10.17 | 7.84
Regression Projection | 9.76 | 7.16
Isomap Deep Regression | 11.01 | 7.56
LLE Deep Regression | 10.05 | 7.43

Fig. 4(a), 4(b), 4(b), and 4(d), are plots of predicted coordinates on the
UJIIndoorLoc dataset. NObLe outputs the most structured prediction compared to
the true floor plan. We see that deep regression outputs are spread out. From
the satellite view in Fig. 1, we know that middle area of the top left
building is not part of buildings; however, a considerable number of the deep
regression outputs lie in this area. Manifold Embedding predicts fewer points
in this area and is visually more structured compared to Deep Regression. This
is as expected because Isomap Embedding is reconstructed with the aim to
approximate output structure. Also, Deep Regression Projection resembles the
building structure because it eliminates prediction based on human-crafted
maps.

(a)

(b)

(c)

(d)

Figure 5: (a) Network architecture for IMU localization. (b) User travel paths
for testing. Color dots represent the sampling position along this path. Gray
dots represents other sampling position in the dataset but not on this path.
Changing of color represents the the travel sequence. (c) and (d) are IMU
predicted coordinates for Deep Regression and NObLe respectively.

On IPIN2016, NObLe achieves an average error distance of 1.13m and a median
average error distance of 0.046m, while the Deep Regression gives an average
error distance of 3.83m. The best mean error distance on the indoor
localization ranking at IndoorLocPlatform website [17] is 3.71m.

###  IV-C Energy Measurement:

We measure energy consumption on the Nvidia Jetson TX2 module. Using
UJIIndoorLoc, the average running energy for each inference is 0.00518J, and
the average latency is 2 milliseconds.

##  V Application : Device Tracking Using IMUs

In this section, we will discuss the detailed system design for device
tracking using IMU signals. A user travels along a certain path, and a
sequence of IMU data corresponding to this travel path is recorded. Given this
sequence, we want to predict the user location at the end of this path.
Without available public datasets, we collect labeled data over an outdoor
space of 160m by 60m ourselves and show that NObLe achieves accurate device
tracking in terms of path ending position errors.

###  V-A Data Collection

We follow the standard setting of device tracking using IMU. We collect our
data from two independent walks around an area of 160m by 60m on our
university campus. The sampling frequency is around 50Hz, and the total
walking time is around 1 hour and 15 minutes. There are in total 177 reference
locations with GPS coordinates (longitude and latitude). Between each
reference point, there are 768 readings for each inertial sensor on a single
axis. We record 3-axis gyroscope, 3-axis accelerometer, and timestamps. We
construct walking path as follows: (1) randomly choose a reference location as
start position, (2) randomly choose a path length less than 50 and determine
the end position accordingly, (3) concatenate IMU readings between starting
and ending positions as the input. In total, we obtained 6857 paths, and we
use 4389 for training, 1096 for validation, and 1372 for testing.

###  V-B System Architecture

The input consists of two parts: (1) initial location coordinates $h_{start}$
and (2) a sequence of IMU signals $G=g_{1},g_{2},\ldots$, where
$g_{i}\in\mathbb{R}^{d\times n}$. $d$ is the dimension of each inertial sensor
readings and $n$ is the number of sensors. We perform output space
quantization at $\tau=0.4$m and assign neighborhood classes $c$ for path
ending location. Following the NObLe formulation, our model takes
$(G,h_{start})$ as inputs, and predicts $\hat{c}$. Then, we calculate ending
position in longitude and latitude based on predicted neighborhood class
$\hat{c}$.

Our system includes three main parts: (1) projection module, (2) displacement
module, and (3) location module. The projection module takes $g_{i}$ and
outputs an embedding in a lower dimension. Then, all projection embeddings are
concatenated together. Each $g_{i}$ is multiplied by the same trainable
projection weight. The concatenated embedding is passed into the displacement
module, a two-layer feed-forward neural network that predicts the displacement
vector of a user’s travel path. This module is not environment-specific, and a
trained module can be plugged into other models designed for location tracking
in other environments. Taken projected embedding, the displacement network
outputs a displacement vector $V\in\mathbb{R}^{2}$ for tracking on the 2-D
plane or $V\in\mathbb{R}^{3}$ for 3-D tracking involving floors. The location
network takes the resulting displacement vector and one-hot encoded starting
location class, and outputs location class at the end of travel path. We used
Xavier initialization [20] and batch normalization [21] for training. The
overview of our system is shown in Fig. 5(a).

###  V-C Performance Evaluation

NObLe achieved a mean error distance of 2.52m and a median distance of 0.4m.
[8] iterative corrects prediction location at all turnings on the path and
achieves an average error distance of 4.3m. LocMe [19] reports a median of
1.1m position error on test-bed size of 70m by 100m by constantly correcting
at elevators and walls. We could not test their method on our dataset as they
did not open source their code. It is evident that incorporating of map
knowledge is essential in these two previous works. However, both of these
systems require human effort to transfer map knowledge into heuristic rules.

Similar to our experiment on Wi-Fi Localization, we implemented Deep
Regression in order to demonstrate NObLe’s structure awareness. The results
are shown in Table III.

TABLE III: Position error distance (m) for IMU tracking. | Mean | Median
---|---|---
Deep Regression Model | 10.41 | 10.05
[8] | 4.3 | n/a
NObLe | 2.52 | 0.4

Fig. 5(b) shows information on the IMU dataset. From Fig. 5(c), it is evident
that Deep Regression performs poorly on estimating the structural knowledge of
the space since the predicted locations, blue dots, are scattered in the
space. In contrast, NObLe performs better in capturing the structural
information since the predicted location points more closely resembles the
space structure as seen in Fig. 5(d) (cf. Fig. 5(b)).

###  V-D Energy Measurement

We measured energy consumption on an edge computing device emulator, Nvidia
Jetson TX2 module. For a testing path for around 8 seconds, NObLe consumed
around 0.08599J for inference calculation with a 5 milliseconds latency.
Inertial sensors’ energy cost is 0.1356J for 8 seconds, and the total energy
consumption is approximately 0.22159J, which is 27$\times$ less than the GPS
energy requirement 5.925J based on [8].

##  VI Conclusion

We propose a novel method for accurate localization and device tracking
problem, Neighbor Oblivious Learning (NObLe), with the focus on the structure
of the output space. We demonstrated that our formulation is essentially
equivalent to manifold learning without approximation of local Euclidean
distances in the input space. We applied NObLe on two orthogonal applications,
Wi-Fi localization and IMU tracking, and showed a significant increase in
localization accuracy.

## References

  * [1] Markets And Markets, ”Location Analytics Market by Component (Solutions and Services), Location Type (Indoor Location and Outdoor Location), Application (Remote Monitoring, Risk Management), Vertical (Retail, Government and Defense), and Region - Global Forecast to 2025”, https://www.marketsandmarkets.com/Market-Reports/location-analytics-market-177193456.html, 2020,
  * [2] T. O. Oshin, S. Poslad, and A. Ma, ”Improving the Energy-Efficiency of GPS Based Location Sensing Smartphone Applications”, 2012 IEEE 11th International Conference on Trust, Security and Privacy in Computing and Communications, 2012, 99.1698-1705
  * [3] P. Bahl and V. N. Padmanabhan, ”RADAR: an in-building RF-based user location and tracking system,” Proceedings IEEE INFOCOM 2000. Conference on Computer Communications. Nineteenth Annual Joint Conference of the IEEE Computer and Communications Societies (Cat. No.00CH37064), Tel Aviv, Israel, 2000, pp. 775-784 vol.2
  * [4] A. M. Ladd, K. E. Bekris, A. P. Rudys, D. S. Wallach and L. E. Kavraki, ”On the feasibility of using wireless ethernet for indoor localization,” in IEEE Transactions on Robotics and Automation, vol. 20, no. 3, pp. 555-559, June 2004
  * [5] J. Yi, J. Zhang, D. Song, and S. Jayasuriya, ”IMU-based localization and slip estimation for skid-steered mobile robots,” 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems, San Diego, CA, 2007, pp. 2845-2850
  * [6] Q. Yuan and I. Chen, “Localization and velocity tracking of human via 3 IMU sensors,” Sensors and Actuators A: Physical, Volume 212, 2014, Pages 25-33, ISSN 0924-4247
  * [7] M. A. Youssef, A. Agrawala and A. Udaya Shankar, ”WLAN location determination via clustering and probability distributions,” Proceedings of the First IEEE International Conference on Pervasive Computing and Communications, 2003. (PerCom 2003)., Fort Worth, TX, 2003, pp. 143-150
  * [8] E. J. J. Gonzalez et al., ”Location detection for navigation using IMUs with a map through coarse-grained machine learning,” Design, Automation & Test in Europe Conference & Exhibition (DATE), 2017, Lausanne, 2017, pp. 500-505
  * [9] X. Wang, L. Gao, S. Mao and S. Pandey, ”CSI-Based Fingerprinting for Indoor Localization: A Deep Learning Approach,” in IEEE Transactions on Vehicular Technology, vol. 66, no. 1, pp. 763-776, Jan. 2017
  * [10] M. Abbas, M. Elhamshary, H. Rizk, M. Torki and M. Youssef, ”WiDeep: WiFi-based Accurate and Robust Indoor Localization System using Deep Learning,” 2019 IEEE International Conference on Pervasive Computing and Communications (PerCom, Kyoto, Japan, 2019, pp. 1-10
  * [11] X. Song et al., ”A Novel Convolutional Neural Network Based Indoor Localization Framework With WiFi Fingerprinting,” in IEEE Access, vol. 7, pp. 110698-110709, 2019
  * [12] J. Torres-Sospedra et al., “UJIIndoorLoc: A New Multi-Building and Multi-Floor Database for WLAN Fingerprint-Based Indoor Localization Problems,” 2014 Int Conf Indoor Position Indoor Navigation Ipin, pp. 261–270, 2014
  * [13] S. T. Roweis and L. K. Saul, “Nonlinear Dimensionality Reduction by Locally Linear Embedding,” Science, vol. 290, no. 5500, pp. 2323–2326, 2000
  * [14] J. B. Tenenbaum, V. de Silva, and J. C. Langford, “A Global Geometric Framework for Nonlinear Dimensionality Reduction,” Science, vol. 290, no. 5500, pp. 2319–2323, 2000
  * [15] M. M. Bronstein, J. Bruna, Y. Lecun, A. Szlam, and P. Vandergheynst, “Geometric Deep Learning,” Ieee Signal Proc Mag, vol. 34, no. 4, pp. 18–42, 2017
  * [16] Y. Bengio, A. Courville, and P. Vincent, “Representation Learning: A Review and New Perspectives.”, International Conference on Learning Representations, 2014
  * [17] J. Torres-Sospedra et al., IPIN2016 Tutorial, http://indoorlocplatform.uji.es/databases/get/2/
  * [18] K. S. Kim, S. Lee, and K. Huang, “A scalable deep neural network architecture for multi-building and multi-floor indoor localization based on Wi-Fi fingerprinting,” Big Data Anal, vol. 3, no. 1, p. 4, 2018
  * [19] Xinye Lin and Xiao-Wen Chang and Xue Liu, “LocMe: Human Locomotion and Map Exploitation Based Indoor Localization,” 2017 IEEE International Conference on Pervasive Computing and Communications (PerCom), pp. 131–140, 2017
  * [20] X. Glorot and Y. Bengio, “Understanding the Difficulty of Training Deep Feedforward Neural Networks,” AISTATS, 2010
  * [21] S. Ioffe and C. Szegedy, “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift,” ICML, 2015
  * [22] H. Liu, H. Darabi, P. Banerjee, and J. Liu, “Survey of Wireless Indoor Positioning Techniques and Systems,” IEEE, 2007


WGMwgm QEqe EPep PMSpms BECbec DEde

[type=editor, bioid=1, ] [1]

[type=editor, bioid=2, ]

[type=editor, bioid=3, ]

[cor1]Corresponding author

#  blackA Systematic Review of Computational Thinking in Early Ages

Edelberto Franco Silva<EMAIL_ADDRESS>Bruno Jose Dembogurski
<EMAIL_ADDRESS>Gustavo Silva Semaan<EMAIL_ADDRESS>
###### Abstract

Nowadays, technology has become dominant in the daily lives of most people
around the world. From children to older people, technology is present,
helping in the most diverse daily tasks and allowing accessibility. However,
many times these people are just end-users, without any incentive to the
development of computational thinking (CT). With advances in technologies, the
abstraction of coding, programming languages, and the hardware resources
involved will become a reality. However, while we have not progressed to this
stage, it is necessary to encourage the development of CT teaching from an
early age. This work will present state of the art concerning teaching
initiatives and tools on programming (e.g., ScratchJr), robotics (e.g., KIBO),
and other playful tools (e.g., Happy Maps) for the development of CT in the
early ages, specifically filling the gap of CT at the kindergarten level. This
survey presents a systematic review of the literature, emphasizing
computational and robotic tools used in preschool classes to develop the CT.
blackThe systematic review evaluated more than 60 papers from 2010 to December
2020, electing 31 papers and adding three papers from the qualitative stage.
The paper’s amount was classified in taxonomy to show CT’s principal tools and
initiates applied to children early. To conclude this survey, an extensive
discussion about the terms and authors related to this research area is
present.

###### keywords:

Early years Improving classroom teaching Teaching-learning strategies 21st
century abilities

##  1 Introduction

In this century, people are interested not only in being consumers of
technology but also in producing them. Differently to the past decades, today,
the children grow up as digital natives. However, to motivate people to not
grow only as an end-user, it is necessary to drive them and offer tools to
develop computational thinking (CT) as soon as possible. The development of CT
is not only essential to create future scientists or engineers, but also it
can enhance many cognitive and intellectual skills, allowing people to solve
real problems as to “find the best path from his/her house to the market”, or
to “calculate the trajectory of an object”.

The CT was credited to Seymour Papert (Papert, S.,, 1980) in 1980, but only in
2006 Jeannette M. Wing (Wing, J. M.,, 2006) popularized the term and sparked
the international community’s interest. Many researches were conducted in the
last decade to understand and propose new strategies to develop the CT (Bers,
M. U.,, 2017; Bers, M.U.,, 2019; Bers, M. U.,, 2019; Palmer, H.,, 2017; Ehsan,
H., Dandridge, T.M., Yeter, I.H., Cardella, M.E.,, 2018). However, a little
part of these initiatives focuses on early ages, creating a gap to apply STEM
education (science, technology, engineering, and math) and develop the CT on
the next generations. A overview and updated about the CT in general can be
found in Yasar et al. (Yasar, O.,, 2017).

black The importance of developing the CT at an early age is discussed by
actual papers and books (Bers,, 2020; Sanford and Naidu,, 2016). Its primary
motivation is to provide a familiarity with CT from an early age,
incorporating the benefits to think logically in real-life challenges.
However, to exist a motivation to propose and evaluate new curricula and tools
from children at an early age, the papers published about this topic are
needy. More than this, it is necessary to investigate the state-of-the-art in
this field and discuss its characteristics and benefits.

black To fill this gap, in this paper, we introduce a systematic review of
curriculum initiatives and tools applied to CT from an early age. This work is
the first to present the state-of-the-art in this field of knowledge, focusing
only on kindergarten and children with 2-5 years old. To conduct our research
was considered the decade from 2010 to 2020. Besides presenting the most
relevant papers published in the last decade, a growth trend analysis of this
research field is introduced as a taxonomy of the main tools used to develop
CT at an early age.

A systematic review has clear the emergence of this research topic, showing an
increasing number of published papers, a growing of more than three times in
the last five years. The proposal of new curriculum and the development of new
tools to support them topics that needs to be discussed, since a significant
number of these are playable using robotic and tangible parts. Another
relevant topic is the prevalence of papers in journals concerning conferences,
creating the necessity to investigate this topic on symposiums and scientific
events. Lastly, the number of papers with case studies and surveys is twice
compared to paper without them, reinforcing the need for validation and
feedback by education researchers.

Figure 1: The word cloud shows each word’s relative relevance based on the
systematic review papers.

In Figure 1 is shown the most common words across all papers: computational,
thinking, programming, and robotics. As aforementioned, we will see in this
article, the main initiatives are related to robotics and programming as a way
to create playful tools and incentivize the CT at an early age. The word cloud
is vital to show the main words and their relevance – where more bigger words
represent more occurrences on the selected papers than small ones. blackIt is
relevant to comment about the limitation of this work. In this paper we only
consider the gap of CT for the kindergarten level concerning teaching
initiatives and tools on programming, robotics, and other playful tools. We do
not consider the impact of each initiative applied from an early age to the
rest of life (i.e., young and adult life).

blackThus, the main question to be answered by this article is: “Which tools
and strategies were adopted to encourage the CT development at early ages?”

The remainder of this article is structured as follows: section 2 presents the
methodology approach used on this paper, and how the papers were selected;
Section 3 describes the leading tools, curriculum initiatives and classify
than following a taxonomy proposed by the authors; Section 4 conclude this
article suggesting a debate and possible future works.

##  2 Methods - Systematic Review

In this work, a mapping of state of the art was conducted based on a
systematic review. This method is responsible for guiding us about collecting
the most relevant papers in the last decade, respecting the keywords chosen to
answer the question created on the protocol step. The next paragraph explains
step by step how we constructed the base of this article. Kitchenham and
Charters (Kitchenham and Charters,, 2007) outline three phases to conduct an
systematic review: (i) planning; (ii) execution and (iii)
summarization/reporting. In the first phase, it is necessary to identify the
need for a review and create a review protocol containing the systematic
review’s important information. The second phase identifies and selects
relevant primary studies, performs the data extraction, and synthesizes the
extracted data. Finally, in the third phase, the systematic review results are
summarized and published to the community. blackSimilar to (Kitchenham and
Charters,, 2007), the PRISMA 2020111http://www.prisma-statement.org/ (Moher et
al.,, 2011) statement and its checklist and flow diagram. All steps were
conducted using the software StArt (Fabbri et al.,, 2016) that implements all
requirements described.

Four indexed libraries were selected, namely:

  * •

Scopus - Elsevier

  * •

ACM

  * •

IEEE

  * •

Web of Science

Figure 2: The phases of systematic review following the blackKitchenham and
Charters (Kitchenham and Charters,, 2007) and PRISMA (Moher et al.,, 2011)
guidelines.

Using the logical keywords combination and restricting the search only to
articles and proceedings, written in English and the range of black from
January 2010 to December 2020 , the selection was started. The search string
used on all libraries follows the logic: ((‘‘computational think*’’) AND
((‘‘early age’’) OR (‘‘kindergarten’’) OR (‘‘preschool’’)) AND ((‘‘playful
tool’’) OR (‘‘programming’’) OR (‘‘robotic’’))), beyond the restrictions
already mentioned.

Thus, 72 papers were recovered in the selection step, where 13 papers were
excluded as duplicate entries, and 20 rejected them not to be in the scope of
“early ages” (dealing with children older than five years). With the 39
selected papers, the extraction stage was initiated. In this step, 31 papers
were chosen, based on the exclusion of 8 articles because they are outside the
standards of full articles in English. Finally, three relevant papers were
included in the qualitative inclusion stage, totaling 34 papers.

With this final selection, three were identified having a general review
approach to the concept of CT at in early age, seven deal with the proposal
for the new curriculum, and 24 modern tools, with or without case studies. It
is interesting to note that the case study proves to be of great value in this
context and of all works, where 22 articles of the total amount present
validation per case study - which represents 6̃4% of the papers. Figure 2
shows the steps described in this section, as well as the proportion of paper
with (Yes) or without (No) use cases in Figure 4.

Figure 3: Number of papers published on conference versus journals. Figure 4:
Distribution of paper with (Yes) or without (No) use cases. Figure 5: The
graph shows the cumulative number of papers per year and its growth trend.

It is noteworthy that the majority number of papers on journals than
conferences, as we can see in Figure 3. This kind of behavior is not observed
in many other research fields. It creates a feeling about the necessity to
encourage all people involved in CT to create more forums to discuss the
topic, e.g., on symposiums or conferences.

Another relevant preliminary analysis is the relevance of use cases. The
majority of the papers present a kind of validation, as through surveys or
observations. The distribution of papers with or without use cases can be seen
in Figure 4.

As previously mentioned, we will present the main initiatives for the
development of CT at an early age. This article is the first research that we
know that addresses the topic precisely at an early age. However, we can see
that the number of studies and publications has grown in recent years,
according to Figure 5. The last five years are more than three times bigger
than the other years. It is possible to see the growth trend of this research
field.

This section introduced how the systematic review was conducted and showed the
main characteristics of the papers and their trends. Thus, we can present all
papers in the next section.

##  3 Results

The following sections will present the main existing tools, guiding the
reader on the selected papers the articles with the proposals and evaluations
of the school curriculum, and describing a taxonomy for all of the other
papers.

###  3.1 Tools

As part of the initiatives and efforts to develop computational thinking in
early ages, tools such as robotics and computer programming initiatives are
increasingly being encouraged among researchers and early childhood educators.
Robotics and computer programming in this context can support a range of
cognitive and social aspects. We will introduce the leading technology
solutions that were cited in the selected researches. Basically, in the early
ages, there is a great incentive to use block-based programming, a playful way
to encourage the child. There is also an incentive to use small robots that
memorize simple commands like moving forward, side or back, to create a
spatial vision and cognition capacity. There are web-based tools, as we will
see, but these are more complex for children at an early age, and there are
block solutions that use both the tangible form and the aid of a device such
as a tablet or iPad.

As a way of organizing the presentation and tools’ classification, Figure 6 is
presented where it is possible to see BeeBot, Kibo, ScratchJr, and a group of
several other tools that appear less frequently in the verified studies. Thus,
we will present each one of them below.

Figure 6: Percentage of main tools occurrence.

####  3.1.1 Bee-Bot

The Bee-Bot222http://bee-bot.us is a simple robot based on simple commands
focusing on early ages children. This tool is capable of improving the
children’s skills and develop the CT at an early age, only using a couple of
sequences. Their main idea is to enhance the cognition capacity and future
view of steps through directional language as a programming sequence: forward,
backward, left, and right 90 degree turns. In Figure 7 is possible to see the
robot and their commands on top of then. Thus, the sequence is created by the
child before to click on the start. After it, the tool enables his/her to run
all commands sequentially storage.

Figure 7: A general view of Bee-Bot with their commands on the top.

This robot works as a playful tool once it improves the CT skills through the
necessity to memorize the steps and imagine how the robot will run the
actions. It is possible to create a puzzle for children using a square paper
or a map with obstacles. An example of this kind of problem is shown in Figure
8.

Figure 8: An example of Bee-Bot map.

####  3.1.2 KIBO

The KIBO robotics kit is a tool (Sullivan, A., Elkin, M., Bers, M. U.,, 2015)
that propose to engage young children in both building and programming. The
KIBO is a kit developed by the DevTech Research Group at Tufts
University333https://sites.tufts.edu/devtech/research/kibo-robot/ and
commercialized by KinderLab Robotics444https://kinderlabrobotics.com/, an
enterprise created with a focus do enhance the STEM. Although the KIBO
designed for young children with ages from 4 to 7 to learn foundational
engineering and programming content, it can be applied to help do develop CT
at an early age. Figure 9 is possible to see its main components, the robot,
and its blocks with several actions. Once the blocks are read, the robot can
run the steps sequentially. It is possible to create another logical
programming with the blocks beyond to go forward or change direction; one of
these examples is the loop block, mainly used to develop procedures more
complex. Melin et al. (Elkin, M., Sullivan, A., Bers, M. U.,, 2016) present a
case use with children with three years old, where the tool is used to
validate a proposal to incorporate than on an urban public preschool in Rhode
Island, US.

Figure 9: KIBO robot tool with the main blocks reading the code bar by
infrared (Elkin, M., Sullivan, A., Bers, M. U.,, 2016).

####  3.1.3 ScratchJr

The ScratchJr555https://www.ScratchJr.org/ is a programmable tool web-based
free of charge that can be used since the early ages. But this solution is not
so simple than KIBO or Bee-Bot needing more help from an adult to understand
and manipulate the computer/tablet. Even focusing on children with 5-7 years
old, it is common to find many use cases to adopt this tool.

Its motivation is to improve logic programming from an early age. Like its
version for young people, this tool has the purpose of creating funny stories
using a sequence of blocks. In Figure 10 is possible to see its web interface
where a chain of blocks is put on the way to create the story. Although to be
more complex, it is possible to create more simple stories with a limited
number of blocks, making it possible to be applied at an early age.

Figure 10: ScratchJr.

####  3.1.4 Others

Other tools are cited by many papers, since prototypes of robots to tangible
and web-based block-based solutions. Examples of robotic tools are LEGO WeDo
and EV3, both are solutions of LEGO company, and their objective is to
evaluate the CT for STEM. Despite being on the field of this work, we consider
both of them more indicated to be incorporated in the curriculum for children
with more than eight years old because of the necessity to manipulate small
parts and have the necessary knowledge to manage the computer. Other papers
cited the mBot, but it is more like LEGO WeDo. Differently of the tools
mentioned, other work proposes a prototype of a playful robotic tool called
Robotito. This initiative follows the same KIBO concepts, but today it is only
in an initial stage. Another new robot is BlueBot that does the same things as
Bee-Bot, but with Bluetooth connection support. Other initiatives as LightBot
follow the same of ScratchJr, doing possible to programming with blocks on a
web-based application, and can be used for children with five years, for
example. More interest to the early ages in this section is the Happy Maps, an
unplugged activity. The children can create simple algorithms as a set(s) of
instructions to move a character through a maze using a single command. Figure
11 shows an example of a maze that can be solved by arrows that indicate how
the character finds the fruit.

Figure 11: Happy map with a simple example of puzzle to solve a maze - Source:
http://code.org.

They are a diverse number of tools and prototypes in the literature, and it
does not enhance the discussion at this moment. About it, the other tools will
be suppressed at this moment. If a tool that was not introduced in this
section and appeared ahead, it will be presented in its context.

After knowing about the leading tools in the literature, it is possible to
introduce the papers where a curriculum strategy was proposed or validated.

###  3.2 Curriculum

Many countries have taken initiatives to introduce CT and STEM skills into the
classroom, changing the curriculum mainly for people since the K-10 or K-11.
It is possible to mention a couple of examples. But in this work, we are
focusing since 2 to 8 years old. In the US, the Computer Science For All was
created in 2013, to teach students from childhood to high school computer
science. In 2014, the European Commission sent a joint letter to the EU
Education Ministers, urging them to promote computer programming. In 2012, The
Royal Society published a report showing an unsatisfactory result about CT
skills development initiatives, proposing a curriculum remodeling. The
consequence was the incorporation of the new subject “Computing” in primary
school (children from 5 to 7 years). France has announced the introduction of
an optional programming course in primary schools. Finland and Italy are
considering codification initiatives for young people. In Spain, many efforts
are being conducted as a non-profit initiative called Programamos, whose
fundamental objective is to promote CT’s development from an early age through
the programming of video games and applications in all school stages. To
illustrate it, a set of papers dedicated to propose or validate a curriculum
will be present below.

In Bers et al. (Bers, M.U.,, 2010) was present for the first time the
TangibleK, complemented by another paper published in 2014 (Bers, M. U.,
Flannery, L., Kazakoff, E. R., Sullivan, A.,, 2014). TangibleK is an education
program that uses robotics as a tool to engage children in developing
computational thinking and learning about the engineering design process.
Supported by the NSF (National Science Foundation), the educational program
emerged in 2010 as a perspective for the insertion of computing and CT for
children in kindergarten. In (Bers, M. U., Flannery, L., Kazakoff, E. R.,
Sullivan, A.,, 2014) was conducted a use case considering the five years of
learning since the education program’ proposal. The program intends to be a
guide/framework to assist in developing CT, without being tied to robot
technologies or programming languages. The authors highlight the project’s
results ans its great innovations and help to change the curriculum in
schools. Responsible to conduct this research, Prof. Marina Umaschi Bers, from
Tufts University666https://sites.tufts.edu/devtech/ describe that program as a
research involving three dimensions: theoretical contributions, design of new
technologies, and empirical work to test and evaluate the theory and the
techniques. This research group is the same responsible to create programming
languages such as KIBO and ScratchJr, as well as teaching materials and
pedagogical strategies for early childhood educators’ professional development
and community engagement.

Palmer (Palmer, H.,, 2017) presents a teaching intervention carried out over
four months in two Swedish preschools, where programming was used to
facilitate preschoolers’ learning of mathematics, especially in their
development of spatial thinking. This study focuses on children between 3 to
4.5 years old, five girls and three boys. Although its small number of
participants, the methodology adopted can be considered a dominant feature.
The study contains three steps: pretest, intervention, and posttest. The
pretest focusing on instruct the child to know about to follow instructions
about how to move in a room (i.e., walk three steps in front and turn left);
the intervention is composed of four steps for four weeks. The posttest is
conducting in rounds after each step, opening an opportunity of each child
talks about was learned.

In Ehsan et al. (Ehsan, H., Dandridge, T.M., Yeter, I.H., Cardella, M.E.,,
2018) is present a research as part of the project - supported by NSF - that
analyzes the PictureSTEM curriculum777http://picturestem.org/. The authors
consider the K-12 STEM to incorporate CT in pre-college education. In this
way, the authors look for the ways K-2 children engage in CT in school and
out-of-school settings, applying a theoretical framework to find and help CT’s
development. After conducting the case study, the authors conclude that it
provides evidence that children can engage in CT competencies in different
problem-solving contexts, including STEM, particularly engineering.

Munoz-Repiso et al. (Munoz-Repiso, A. G., Caballero-Gonzalez, Y.,, 2019)
evaluate the repercussion of educational robotics activities on kindergarten
students in the acquisition of CT and programming skills. The research design
is quasi-experimental, with pre-test and post-test measures, using
experimental and control groups. The sample consists of 131 students from the
second cycle of early education (between 3 and 6 years old), all from the same
Spanish school in Salamanca’s city. This initiative is based on the reference
program of robotics studies TangibleK.

Recently, Prof. Marina Umaschi Bers, one of the most enthusiastic of CT
development at an early age, presents in (Bers, M. U.,, 2019) a pedagogical
approach for teaching computer science in early childhood. This paper
describes an educational strategy for early childhood computer science called
“Coding as Another Language” (CAL), six coding stages, or learning
trajectories that young children go through when exposed to the CAL
curriculum. CAL is grounded on the principle that learning to program involves
learning how to use a new language (a symbolic system of representation) for
communicative and expressive functions. Besides indicating the methodology and
how the activities can be incorporated into the curriculum, case studies of
young children using either the KIBO robot or the ScratchJr were used to
characterize each coding stage and illustrate the instructional practices of
CAL curriculum.

In Nam et al. (Nam, K.W., Kim, H.J., Lee, S.,, 2019) was examined the effects
of a card-coded robotics curriculum and associated activities on
kindergarteners’ sequencing and problem-solving skills. Kindergarteners
participated in card-coded programming using a robot called
TurtleBot888https://www.turtlebot.com/. A card-coded robot curricular
intervention was also designed to enhance their planning behaviors using
complementary tools. This study examined an 8-week robotic curricular
intervention through assessment of 53 participants ranging in age from 5 to 6
in South Korea, while also evaluating sequencing and mathematical problem-
solving in both the treatment and comparison groups. It was found that
children in the treatment group who engaged in the card-coded robotic
curricular intervention performed better on sequencing and problem-solving
tests.

###  3.3 Review

This section is introduced to show some related work about surveys/evaluation
using tools or CT curriculum proposals. Considering the scope of this paper,
nor have a high number of related work in this area, it is possible to see
surveys and studies about tools and CT strategies in pre-schools. As well as
the ”Curriculum” group, this section is considered an external box at taxonomy
proposed but keeps its correlation with the research theme.

Umam et al. (Umam, M.U.K., Budiyanto, C., Rahmawati, A.,, 2019) reviewed the
advantages and disadvantages of commercially available robotics devices. The
study shows a critical analysis method to the literature published from 1952
to 2017 in international journals and proceedings. Although only three robotic
solutions were investigated, and the discussion is limited to show an overview
of the tools.

In Rich et al. (Rich, P.J., Browning, S.F., Perkins, M.K., Shoop, T.,
Yoshikawa, E., Belikov, O.M.,, 2019) the authors carry out a 20-minute
snowball study with groups that teach disciplines related to computing in
schools. Around 300 teachers and 60,000 primary school students were
interviewed (less than 5 years and up to 14 years, separated into groups 2
years apart). Participants are mainly from the USA and the UK, but also other
countries. The most used tools at an early age were: ScratchJr, KIBO, and
Unplugged Activities. The authors reach conclusions with their research as:
(i) students are capable; (ii) it is ok to fail; (iii) there are a lot of
resources to teach computing; (iv) start simple and teach basics; (v)
unplugged activities is a good strategy.

Differently, Esteve-Mon et al. (Esteve-Mon, F.M., Adell-Segura, J., Nebot,
M.A.L., Novella, G.V., Aparicio, J.P.,, 2019) present a study to investigate
how to prepare future teachers of Kindergarten and Elementary School in the
educational use of CT. Their proposal was conducted on the field of
educational robots that includes unplugged, playing, making, and remixing
activities. A survey was conducted with 114 Spanish university students of
education. The authors divide the strategy into four steps: (i) unplugged,
e.g., cut and paste (ii) test of the robots, e.g., Bee-Bot; (iii) activities
to ScratchJr individual and group tasks; (iv) and a final mission to resolve a
challenge.

###  3.4 Taxonomy

In this section, we propose a taxonomy that can be seen in Figure 12. There
are 3 main categories, curriculum, review, and tools. The review and
curriculum have already been presented, and this section emphasis on the
tools. In tools, there are four possible types: block, robot, block and robot,
and prototype. Whenever there is a tool based on blocks, it may be tangible or
web. Despite having the taxonomy organized that way, it is possible to have
papers only classified in one, another, or more than one kind of tool.

Figure 12: Proposed taxonomy.

####  3.4.1 Block

Web

In Papadakis et al. (Papadakis, S., Kalogiannakis, M., Zaranis, N.,, 2016) a
case study carried out in 2016 on the ScratchJr tool is presented. The study
sample consisted of 43 preschool children (22 boys, 21 girls) who were
attending classes in public and a private kindergarten in the region of Crete,
Greece, during the school year 2014–2015. Findings reveal that ScratchJr
enhances student interest by making the learning experience fun. Similarly,
animated scenarios showed high levels of engagement among students.
Specifically, ScratchJr allowed children to engage in deep reflection as they
solved problems and collaborated with their peers, both of which activities
enhanced their learning experience. Part of the authors from (Papadakis, S.,
Kalogiannakis, M., Zaranis, N.,, 2016) published another paper in the same way
in (Papadakis, S., Kalogiannakis, M.,, 2019). This new paper focused on the
future teacher of kindergarten, to create the CT on the people will learn in
the future for the kids. The authors adopted Scratch as the introductory
programming language for a semester in the Department of Preschool Education
in the University of Crete. The aim of using Scratch was to excite students’
interest and familiarise them with the basics of programming. For 13 weeks,
students were introduced to the main Scratch concepts and were asked to
prepare their projects afterward. For the projects, they were required to
develop a game to teach specific concepts about Mathematics or Physical
Science or present an Aesop myth to preschool age students. The results we
obtained were more satisfactory than expected and, in some regards,
encouraging.

The authors of (Lowe, T.A., Brophy, S.P.,, 2019) looks at the intersection of
CT and computer science in first-grade learners who are developing
computational solutions involving literacy tasks. Students retell a story by
animating characters in ScratchJr by breaking down the story, creating an
animation storyboard, and finally implementing the plan in ScratchJr. For most
of the participants, this is their first time using ScratchJr or any
programming language. Therefore, their early experience with technology means
they are working on an analysis of a story using literacy skills, considering
a visual representation of the story, and learning how to realize the story
line’s expression using a computer language. Despite to did not be directly
applied to the early age children, it is an excellent motivation for
kindergarten teachers.

In Ciftci et al. (Ciftci, S., Bildiren, A.,, 2020) the researchers evaluated
an experimental study carried out to put forth the impact on the problem
solving and cognitive abilities of computer programming courses applied on
4-5-year-old preschool children. This study uses a pretest-posttest control
group experiment model. According to their results, there is an increase in
the non-verbal cognitive abilities of children in the experiment group with no
statistically significant problem-solving skills. The ‘course A’ from code.org
was applied to the evaluation, with the help of unplugged activities (e.g.,
happy maps) and programming block-based.

Tangible and Web

In Clarke-Midura et al. (Clarke-Midura, J., Lee, V. R., Shumway, J. F.,
Hamilton, M. M.,, 2019), the authors examine 3 block-based coding tools
applying a framework developed based on Gibson’s theory (Adolph, K., Kretch,
K.S.,, 2015) of affordances and Palmer’s external representations (Palmer,
S.,, 1978). They intend to verify known tools for children in early ages,
motivated by more work and solutions for the development of computational
thinking only for children in K-12. It is considered an interesting concept,
which facilitates the child’s interaction with the program, the TUI (Tangible
User Interface) The tools compared are: ScratchJr, Osmo Coding
Awbie999https://www.playosmo.com/en/coding/ and KIBO.

####  3.4.2 Robot

In Gonzalez et al. (González, Y.A.C., Muñoz-Repiso, A.G.V.,, 2017), the
authors show three different robotic technology applied to early school age.
There are six groups of students, with a total of 131 students and eight
teachers distributed in first, second, and third kindergarten. With the ease
of access to groups of students and teachers, the total population will be
used for the study. On the conclusions, the authors received positive results
regarding the acceptance and motivation to use educational activities mediated
by programmable robots in students.

Roussou et al. (Roussou, E., Rangoussi, M.,, 2020) present a research based on
a case study that investigates the impact of robotics on the cultivation of CT
skills in early childhood through an educational intervention implemented in a
typical public kindergarten in Athens, Greece. On the material, the authors
adopt the tool Code & Go Robot Mouse Activity Set from Learning
Resources101010https://www.learningresources.com/code-gor-robot-mouse-
activity-set, a similar to Bee-Bot tool. The authors investigate using a
pretest, intervention, and posttest, proving the benefits of robotics in
kindergarten.

####  3.4.3 Block and Robot

Tangible

In Urlings et al. (Urlings, C. C., Coppens, K. M.,Borghans, L.,, 2019), sixty-
five kindergarteners received assignments to go through a maze with a
programmable robot, the Bee-Bot. The authors conducted this study via
observation, quantifying which time and errors occurred, measuring how it was
the increase of comprehension and evolution from a child with mean age equals
to 6. The results were satisfactory, with a low rate of errors and a
significant increase in the child’s motivation.

Bers et al. (Bers, M.U., González-González, C., Armas-Torres, M.B.,, 2019)
evaluated a “coding as a playground” experience in keeping with the Positive
Technological Development (PTD) framework with the KIBO robotics kit,
specially designed for young children. The research was conducted with
preschool children aged 3–5 years old (N = 172) from three Spanish early
childhood centers with different socio-economic characteristics and teachers
of 16 classes. Results confirm that it is possible to start teaching this new
literacy very early (at three years old). The results show that the strategies
used promoted communication, collaboration, and creativity in classroom
settings. Teachers had excellent experience-enhancing his/her motivation and
confidence to motivate the CT on the students. The concept of “coding as a
playground” as a new literacy, a new language for children where they can
learn to code at a young age through fun, play, and creativity (Bers, M. U.,,
2017).

Web

In Strawhacker et al. (Strawhacker, A., Lee, M., Bers, M. U.,, 2018), the
authors investigated the little is known about the relationship between a
teacher’s unique instructional style and their students’ ability to explore
and retain programming content. The study focuses on children aged 5-8 years.
In this mixed-methods study, quantitative and qualitative data were collected
from 6 teachers and 222 kindergartens through second-grade students at six
schools across the United States from 2 months in 2014. All participants
engaged in a minimum of two lessons and a maximum of seven lessons using the
ScratchJr programming environment to introduce coding. Teachers reported on
their classroom structure, lesson plan, teaching style, and comfort with
technology.

Tangible and Web

In Rial-Fernández et al. (Rial-Fernández, B., Santacruz-Valencia, L. P.,,
2019), the authors describe the case study carried out in a classroom of Early
Childhood Education, with students of 5 years of age, with which basic
programming concepts have been worked out. The results obtained indicate that
they have been able to master the new vocabulary, assimilate the concepts, and
work for themselves with the chosen tool to carry out the intervention. The
tools evaluated were: Bee-Bot, Happy Maps, and another sheet with arrows that
they have to cut out. All tools were applied to be a playful tool using blocks
to increase the CT in programming.

In Otterborn et al. (Otterborn, A., Schonborn, K. J., Hulten, M.,, 2020), it
was conducted systematically investigates how Swedish preschool teachers
implement programming activities in their teaching practice. Data were
collected through a national online survey with 199 participants. Findings
revealed a range of apps and resources used in combination with tablets, where
activity integration takes place as unplugged programming, digital
programming, or a combination of the former. On the survey, the most tools
cited by the teachers of kindergarten were: Bee-Bot, Blue-Bot,
LightBotJr111111https://lightbot.com/, and ScratchJr. Another relevant result
is that the study showed that nine preschool teachers use unplugged
programming to introduce digital programming.

In Kanbul et al. (Kanbul, S. and Uzunboylu, H.,, 2017), the aim is to reveal
the importance of coding education and robotic applications for achieving
21st-century skills in North Cyprus. The skills from pre-school level mapped
by the authors to help in CT development are: (i) Skills: putting in order,
separating into little pieces, giving order; (ii) Software: ScratchJr,
Code.org, the first two periods of Kodable, The foos; (iii) Robotic tools:
BeeBots and KIBO. In the last, the authors strongly recommend the use of tools
from early ages to increase de CT.

In Pugnali et al. (Pugnali, A., Sullivan, A., Bers, M. U.,, 2017) is
investigated the impact of the interface to help and adoption of technology
tools in CT. Children from 4 to 7 years old participated in the survey,
evaluating ScratchJr at iPad and KIBO as the tangible tool. Results suggest
that type of user interface does have an impact on children’s learning but is
only one of many factors that affect positive academic and socio-emotional
experiences. Tangible and graphical interfaces each have qualities that foster
different types of education.

####  3.4.4 Prototype

Robot

In (Tejera, G., Amorin, G., Sere, A., Capricho, N., Margenat, P., Visca, J.,,
2019) the authors present a prototype solution called Robotito, which
comprises the development of programming and robotics skills. The proposal has
a more technical bias and uses ROS (Robot Operating System) standard
mechanisms. Despite the work elucidating computational thinking as a focus,
its presentation is more technical than discussing the impact of the solution
on learning at an early age.

The authors of (Coiro, F., Solis, M.A., Nettle, C.J., Chila, A.,, 2020)
propose an open-source robot platform called Pro-Robot. They aim to foster CT
abilities on preschool children, including a first approach on the built
platform, an embedded processing unit for not requiring any further equipment,
and work on development to minimize costs. The paper is more technical but
very important to increase the options of robot solutions for CT.

Block / Tangible

In Wang et al. (Wang, D., Wang, T., Liu, Z.,, 2014) the authors present an
economic tangible programming tool called T-Maze for children aged 5-9 to
build computer programs in maze games by placing wooden blocks. Through the
use of computer vision technology, T-Maze provides a live programming
interface with real-time graphical and voice feedback. However, as highlighted
by the authors, the focus is on children aged 5-9 years, which imposes
difficulties to be used by children in early ages.

Block / Web

Koracharkornradt in (Koracharkornradt, C.,, 2017) presents a programming game
called Tuk Tuk that aims to assist in the development of CT in children in
kindergarten (a junior version of the game). It allows the child to organize
the blocks to create the steps of a car until they complete a race, accumulate
points, and complete a detailed task. The idea is to develop an understanding
of algorithms. There is the version for older children, a rating similar to
ScratchJr and Scratch.

In Kanaki et al. (Kanaki, K., Kalogiannakis, M.,, 2018) present the
computational environment
PhysGramming121212http://physgramming.edc.uoc.gr/programming_en.html, which
was designed to be used by children of early childhood age, between 4 and 8
years old. PhysGramming deploys an hybrid schema of visual and text-based
programming techniques, with emphasis on object-orientation, in order to
introduce elementary programming concepts in early childhood education. The
solution can provide three kinds of games: puzzles, matching games and group
games. In each game, instructions about its functionality are given through
the use of an animated cat. An example of application is the teacher creating
digital games in order to help the students learn the names of some animals of
the jungle, together with their nutritional habits. The authors have the
ambition to present basic concepts of object orientation programming in a
playful way. Despite being a recent prototype, tools in the same vein are
excellent candidates for help in the development of the CT.

Baratè et al. (Baratè, A., Ludovico, L.A., Mauro, D.A.,, 2019) show in this
paper a recent evolution of a web prototype conceived initially to teach music
and CT to preschool and primary school learners through a gamification
approach. The software tool, Legato, is based on the metaphor of building
blocks whose characteristics (e.g., position in space, shape, and color) can
be associated with basic music parameters (e.g., pitch, rhythm, and timbre).
Legato is a web app written using standard languages, such as HTML5, CSS, and
JavaScript; besides, it adopts the Web MIDI
API131313https://www.w3.org/TR/webmidi/ to produce sounds. The prototype is
made publicly available for evaluation and uses in an educational context. The
paper did not restrict the age of the child to use it. However, it is a
technical specification. The main objectives induce to be used only for kids
after 5-years old, once it is necessary to manipulate the blocks to construct
the music on a web-based platform.

CodyColor (Klopfenstein, L.C., Delpriori, S., Maldini, R., Bogliolo, A.,,
2019) is a simplified coding game, which takes basic programming instructions
representing movements (e.g. turn left and turn right) and represents them
with color blocks. In contrast to most other coding games, color-coded
programming relies on no symbolic interpretation on part of the player in
order to be approachable by very young players as well. This initiative is
based on Hour of Code CodyRoby, and it is a new version of CodyColor with
massive multiplayer support.

##  4 Discussion

This paper presented state of the art in CT at an early age based on a
systematic review. Considering only children between 2 and 5 years old were
highlight the main tools and curriculum proposals and evaluation. A taxonomy
was introduced to show the most gaps and advances in the last decade
(2010-2020).

Based on the 34 selected works from the Literature Systematic Review, it was
needed to perform a pre-processing step, where data organization, enrichment,
consolidation and formatting are performed. Specifically, the titles of all
works were fragmented in words, occurrence, and next, exclusion of stop-words
(words without semantic relevance) such as conjunction and prepositions. The
terms of interest were unified through a process of radical reduction or
equivalence to the most frequent similar word.

Table 1: Frequency of words most used. Id | Word | Count | (%)
---|---|---|---
1 | Computational | 15 | 44.1
2 | Thinking | 15 | 44.1
3 | Programming | 12 | 35.2
4 | Robotics | 10 | 29.4
5 | Teaching | 8 | 23.5
6 | Coding | 7 | 20.5
7 | Development | 7 | 20.5
8 | Preschool | 7 | 20.5
9 | Learning | 6 | 17.6
10 | Childhood | 6 | 17.6
11 | Teacher | 5 | 14.7
12 | Kindergarten | 5 | 14.7
13 | Early | 5 | 14.7
14 | Children | 5 | 14.7
15 | Educational | 4 | 11.7
16 | Computer | 4 | 11.7
17 | Activities | 4 | 11.7
18 | Education | 4 | 11.7
19 | With | 4 | 11.7
20 | Skills | 3 | 8.8
Table 2: Ranking of most relevant words by year. Word ($\%$) | 2017 | 2018 | 2019 | 2020
---|---|---|---|---
Computational | 50 | 33 | 35 | 25
Thinking | 50 | 33 | 35 | 25
Programming | 50 | 67 | 24 | 0
Robotics | 17 | 0 | 29 | 50
Teaching | 0 | 67 | 29 | 25
Coding | 17 | 0 | 29 | 25
Development | 17 | 0 | 24 | 25
Preschool | 33 | 33 | 24 | 100
Learning | 33 | 33 | 18 | 0
Childhood | 0 | 0 | 29 | 0
Teacher | 17 | 33 | 18 | 0
Kindergarten | 0 | 0 | 0 | 0
Early | 0 | 0 | 24 | 0
Children | 17 | 33 | 0 | 25
Educational | 17 | 0 | 6 | 50
Computer | 0 | 0 | 18 | 25
Activities | 0 | 0 | 24 | 0
Education | 17 | 33 | 6 | 0
With | 0 | 0 | 18 | 0
Skills | 17 | 0 | 6 | 25

Table 1 shows the 20 most frequent words among the titles of the works. In
highlight, with 15 occurrences, the words Computational and Thinking appear in
about 44% of the analyzed papers. In addition, these terms were broken down by
year, starting in 2017. Due to the low amount of publications, the years 2010,
2014 and 2016 were not reported in Table 2. Still based on this table, for
example, the term Preschool is present in all works published in 2020, while
the terms Robotics and Educational, also with high occurrence, appear in half
of the publication titles.

In the pre-processing stage, the consolidation of authors was performed in
order to identify frequent and interesting researchers in the context of the
present work. Due to typos, abbreviations and omissions of part (s) of the
surname, there was a human verification step to decide, in case of doubt, if
two names correspond to the same author. Table 3 lists the authors who have
more than one work among those considered in this research. It is noteworthy
that, although the search did not consider specific authors, six works by
researcher Marina Umaschi Bers (Bers, M.U.) were among the selection made by
the systematic review, which indicates its strong performance and influence in
the area.

black It is clear that this field needs attention, and the last five years
prove that a growing number of research, papers, and new initiatives have
flourished. The significant amount of papers in journals led us to believe it
is an excellent opportunity to motivate the participants to offer conferences
and events where these new finds could be discussed more thoroughly. Another
important observation was the effort conducted by many countries, such as USA
and Spain to incorporate, in kindergarten’s curriculum, the CT as an
improvement for children’s capabilities in a variety of fields in their life.
Finally, the most common way to validate a proposal, tool etc., is to conduct
a use case following pre-test and post-test in different cities and/or
countries.

black This research’s main question (i.e., “Which tools and strategies were
adopted to encourage the CT development at early ages?”) was answered. It is
clear to see that there are many initiatives, however, a huge part of them is
limited in terms of impact on people’s life. Considering it, the authors
suggest conducting more experiments and evaluations with tools and strategies
for CT development at early ages. These future works need to consider the
actual and long-term impacts in the life of these students.”

Table 3: Most frequent authors from papers discussed. Id | Author | Total | $\%$
---|---|---|---
1 | Bers, M.U. | 7 | 21
2 | Kalogiannakis, M. | 3 | 9
3 | Papadakis, S. | 2 | 6
4 | Sullivan, A. | 2 | 6

Furthermore, there is an ongoing discussion about gender disparity in STEM
fields, where man outnumber women (Sullivan, A and Bers, M. U.,, 2019).
Usually, interventions aiming to increase interest of women in science related
fields are performed during high school and collage with subpar results. New
evidence shows that introducing an appropriate curriculum with CT can increase
girls’ interest in engineering.

Another important issue that must be discussed is how the technology will
affect the future of the next generation of kids. One of the primary drivers
of change in the current world is Artificial Intelligence (AI) and automation.
Job characteristics and required skills will change drastically because of it.
These bring us a variety of opportunities related to social and economic
mobility, all that could lead us to a better society. But in order to reap all
this possible benefits we must sow our future generations with this new
knowledge.

The World Economic Forum (Perisic, I,, 2018) discussed how artificial
intelligence is shaking up the job market and two trends were perceived: one
showing the continuous need for tech jobs and skills; and one pointing to
human-centric skills, which directly depends on human qualities. Also, the
impact of AI is not just theoretical anymore and its influence can be observed
across industries and jobs worldwide. There are way more information about
this matter in different types of media, but this discussion is not the center
of this work.

Thus, the aforementioned points reinforce the need for an early introduction
of CT in order to prepare this future generation to a new reality regarding
the job market. To conclude, the authors believe that the first step to
improving the CT at an early age is to start using these resources as soon as
possible.

## Acknowledgement(s)

Blind version

## References

  * Adolph, K., Kretch, K.S., (2015) Adolph, K., Kretch, K.S. (2015).  Gibson’s theory of perceptual learning.  International encyclopedia of the social and behavioral sciences, 10:127–134.
  * Baratè, A., Ludovico, L.A., Mauro, D.A., (2019) Baratè, A., Ludovico, L.A., Mauro, D.A. (2019).  A web prototype to teach music and computational thinking through building blocks.  ACM International Conference Proceeding Series, pages 227–230.
  * Bers, (2020) Bers, M. U. (2020).  Coding as a playground: Programming and computational thinking in the early childhood classroom.  Routledge.
  * Bers, M. U., (2017) Bers, M. U. (2017).  Coding as a playground: Programming and computational thinking in the early childhood classroom.  Routledge.
  * Bers, M. U., (2019) Bers, M. U. (2019).  Coding as another language: a pedagogical approach for teaching computer science in early childhood.  Journal of Computers In Education, 6(4):499–528.
  * Bers, M. U., Flannery, L., Kazakoff, E. R., Sullivan, A., (2014) Bers, M. U., Flannery, L., Kazakoff, E. R., Sullivan, A. (2014).  Computational thinking and tinkering: Exploration of an early childhood robotics curriculum.  Computers & Education, 72:145–157.
  * Bers, M.U., (2010) Bers, M.U. (2010).  The tangiblek robotics program: Applied computational thinking for young children.  Early Childhood Research and Practice, 12(2).
  * Bers, M.U., (2019) Bers, M.U. (2019).  Coding as another language: a pedagogical approach for teaching computer science in early childhood.  Journal of Computers in Education, 6(4):499–528.
  * Bers, M.U., González-González, C., Armas-Torres, M.B., (2019) Bers, M.U., González-González, C., Armas-Torres, M.B. (2019).  Coding as a playground: Promoting positive learning experiences in childhood classrooms.  Computers and Education, 138:130–145.
  * Ciftci, S., Bildiren, A., (2020) Ciftci, S., Bildiren, A. (2020).  The effect of coding courses on the cognitive abilities and problem-solving skills of preschool children.  Computer Science Education, 30(1):3–21.
  * Clarke-Midura, J., Lee, V. R., Shumway, J. F., Hamilton, M. M., (2019) Clarke-Midura, J., Lee, V. R., Shumway, J. F., Hamilton, M. M. (2019).  The building blocks of coding: a comparison of early childhood coding toys.  Information and Learning Sciences, 120(7-8):505–518.
  * Coiro, F., Solis, M.A., Nettle, C.J., Chila, A., (2020) Coiro, F., Solis, M.A., Nettle, C.J., Chila, A. (2020).  Pre-robot: An open-source educational robotics platform for preschoolers.  CEUR Workshop Proceedings, 2564:18–23.
  * Ehsan, H., Dandridge, T.M., Yeter, I.H., Cardella, M.E., (2018) Ehsan, H., Dandridge, T.M., Yeter, I.H., Cardella, M.E. (2018).  K-2 students’ computational thinking engagement in formal and informal learning settings: A case study (fundamental).  ASEE Annual Conference and Exposition, Conference Proceedings, 2018-June.
  * Elkin, M., Sullivan, A., Bers, M. U., (2016) Elkin, M., Sullivan, A., Bers, M. U. (2016).  Programming with the kibo robotics kit in preschool classrooms.  Computers in the Schools, 33(3):169–186.
  * Esteve-Mon, F.M., Adell-Segura, J., Nebot, M.A.L., Novella, G.V., Aparicio, J.P., (2019) Esteve-Mon, F.M., Adell-Segura, J., Nebot, M.A.L., Novella, G.V., Aparicio, J.P. (2019).  The development of computational thinking in student teachers through an intervention with educational robotics.  Journal of Information Technology Education: Innovations in Practice, 18:139–152.
  * Fabbri et al., (2016) Fabbri, S., Silva, C., Hernandes, E., Octaviano, F., Di Thommazo, A., and Belgamo, A. (2016).  Improvements in the start tool to better support the systematic review process.  In Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering, pages 1–5.
  * González, Y.A.C., Muñoz-Repiso, A.G.V., (2017) González, Y.A.C., Muñoz-Repiso, A.G.V. (2017).  Development of computational thinking and collaborative learning in kindergarten using programmable educational robots: A teacher training experience.  ACM International Conference Proceeding Series, Part F132203.
  * Kanaki, K., Kalogiannakis, M., (2018) Kanaki, K., Kalogiannakis, M. (2018).  Introducing fundamental object-oriented programming concepts in preschool education within the context of physical science courses.  Education and Information Technologies, 23(6):2673–2698.
  * Kanbul, S. and Uzunboylu, H., (2017) Kanbul, S. and Uzunboylu, H. (2017).  Importance of coding education and robotic applications for achieving 21st-century skills in north cyprus.  International Journal of Emerging Technologies in Learning, 12(1):130–140.
  * Kitchenham and Charters, (2007) Kitchenham, B. and Charters, S. (2007).  Guidelines for performing systematic literature reviews in software engineering.
  * Klopfenstein, L.C., Delpriori, S., Maldini, R., Bogliolo, A., (2019) Klopfenstein, L.C., Delpriori, S., Maldini, R., Bogliolo, A. (2019).  Codycolor: Design of a massively multiplayer online game to develop computational thinking skills.  CHI PLAY 2019 - Extended Abstracts of the Annual Symposium on Computer-Human Interaction in Play, pages 453–458.
  * Koracharkornradt, C., (2017) Koracharkornradt, C. (2017).  Tuk tuk: A block-based programming game.  IDC 2017 - Proceedings of the 2017 ACM Conference on Interaction Design and Children, pages 725–728.
  * Lowe, T.A., Brophy, S.P., (2019) Lowe, T.A., Brophy, S.P. (2019).  Identifying computational thinking in storytelling literacy activities with scratch jr. (rtp).  ASEE Annual Conference and Exposition, Conference Proceedings.
  * Moher et al., (2011) Moher, D., Altman, D. G., Liberati, A., and Tetzlaff, J. (2011).  Prisma statement.  Epidemiology, 22(1):128.
  * Munoz-Repiso, A. G., Caballero-Gonzalez, Y., (2019) Munoz-Repiso, A. G., Caballero-Gonzalez, Y. (2019).  Robotics to develop computational thinking in early Childhood Education.  Comunicar, 27(59):63–72.
  * Nam, K.W., Kim, H.J., Lee, S., (2019) Nam, K.W., Kim, H.J., Lee, S. (2019).  Connecting plans to action: The effects of a card-coded robotics curriculum and activities on korean kindergartners.  Asia-Pacific Education Researcher, 28(5):387–397.
  * Otterborn, A., Schonborn, K. J., Hulten, M., (2020) Otterborn, A., Schonborn, K. J., Hulten, M. (2020).  Investigating Preschool Educators’ Implementation of Computer Programming in Their Teaching Practice.  Early Childhood Education Journal, 48(3):253–262.
  * Palmer, H., (2017) Palmer, H. (2017).  Programming in preschool-with a focus on learning mathematics.  International Research in Early Childhood Education, 8(1):75–87.
  * Palmer, S., (1978) Palmer, S. (1978).  Fundamental aspects of cognitive representation.  E. Rosch & B. L. Lloyd (Eds.).
  * Papadakis, S., Kalogiannakis, M., (2019) Papadakis, S., Kalogiannakis, M. (2019).  Evaluating a course for teaching introductory programming with scratch to pre-service kindergarten teachers.  International Journal of Technology Enhanced Learning, 11(3):231–246.
  * Papadakis, S., Kalogiannakis, M., Zaranis, N., (2016) Papadakis, S., Kalogiannakis, M., Zaranis, N. (2016).  Developing fundamental programming concepts and computational thinking with scratchjr in preschool education: A case study.  International Journal of Mobile Learning and Organisation, 10(3):187–202.
  * Papert, S., (1980) Papert, S. (1980).  Mindstonns.  New York: Basic Rooks, 607.
  * Perisic, I, (2018) Perisic, I (2018).  How artificial intelligence is shaking up the job market.
  * Pugnali, A., Sullivan, A., Bers, M. U., (2017) Pugnali, A., Sullivan, A., Bers, M. U. (2017).  The Impact of User Interface on Young Children’s Computational Thinking.  Journal of Information Technology Education-Innovations in Practice, 16:171–193.
  * Rial-Fernández, B., Santacruz-Valencia, L. P., (2019) Rial-Fernández, B., Santacruz-Valencia, L. P. (2019).  The teaching of programming is not the future but the present.  In 2019 International Symposium on Computers in Education (SIIE), pages 1–6.
  * Rich, P.J., Browning, S.F., Perkins, M.K., Shoop, T., Yoshikawa, E., Belikov, O.M., (2019) Rich, P.J., Browning, S.F., Perkins, M.K., Shoop, T., Yoshikawa, E., Belikov, O.M. (2019).  Coding in k-8: International trends in teaching elementary/primary computing.  TechTrends, 63(3):311–329.
  * Roussou, E., Rangoussi, M., (2020) Roussou, E., Rangoussi, M. (2020).  On the use of robotics for the development of computational thinking in kindergarten: Educational intervention and evaluation.  Advances in Intelligent Systems and Computing, 1023:31–44.
  * Sanford and Naidu, (2016) Sanford, J. F. and Naidu, J. T. (2016).  Computational thinking concepts for grade school.  Contemporary Issues in Education Research (CIER), 9(1):23–32.
  * Strawhacker, A., Lee, M., Bers, M. U., (2018) Strawhacker, A., Lee, M., Bers, M. U. (2018).  Teaching tools, teachers’ rules: exploring the impact of teaching styles on young children’s programming knowledge in ScratchJr.  International Journal of Technology and Design Education, 28(2):347–376.
  * Sullivan, A and Bers, M. U., (2019) Sullivan, A and Bers, M. U. (2019).  Investigating the use of robotics to increase girls’ interest in engineering during early elementary school.  International Journal of Technology and Design Education, 29(5):1033–1051.
  * Sullivan, A., Elkin, M., Bers, M. U., (2015) Sullivan, A., Elkin, M., Bers, M. U. (2015).  Kibo robot demo: engaging young children in programming and engineering.  In Proceedings of the 14th international conference on interaction design and children, pages 418–421.
  * Tejera, G., Amorin, G., Sere, A., Capricho, N., Margenat, P., Visca, J., (2019) Tejera, G., Amorin, G., Sere, A., Capricho, N., Margenat, P., Visca, J. (2019).  Robotito: Programming robots from preschool to undergraduate school level.  In 2019 19th International Conference on Advanced Robotics (ICAR).
  * Umam, M.U.K., Budiyanto, C., Rahmawati, A., (2019) Umam, M.U.K., Budiyanto, C., Rahmawati, A. (2019).  Literature review of robotics learning devices to facilitate the development of computational thinking in early childhood.  AIP Conference Proceedings, 2194.
  * Urlings, C. C., Coppens, K. M.,Borghans, L., (2019) Urlings, C. C., Coppens, K. M.,Borghans, L. (2019).  Measurement of Executive Functioning Using a Playful Robot in Kindergarten.  Computers in the Schools.
  * Wang, D., Wang, T., Liu, Z., (2014) Wang, D., Wang, T., Liu, Z. (2014).  A tangible programming tool for children to cultivate computational thinking.  The Scientific World Journal, 2014.
  * Wing, J. M., (2006) Wing, J. M. (2006).  Computational thinking.  Communications of the ACM, 49(3):33–35.
  * Yasar, O., (2017) Yasar, O. (2017).  The essence of computational thinking.  Computing in Science Engineering, 19(4):74–82.