# Crowdsourced Network Measurements in Germany: Mobile Internet Experience
from End User Perspective

Anika Schwind∗ ∗University of Würzburg, Insitute of Computer Science,
Würzburg, Germany
<EMAIL_ADDRESS>
+Nokia Networks, Ulm, Germany<EMAIL_ADDRESS>
⋄Tutela Ltd, Victoria, Canada<EMAIL_ADDRESS>Florian Wamser∗
∗University of Würzburg, Insitute of Computer Science, Würzburg, Germany
<EMAIL_ADDRESS>
+Nokia Networks, Ulm, Germany<EMAIL_ADDRESS>
⋄Tutela Ltd, Victoria, Canada<EMAIL_ADDRESS>Tobias Hoßfeld∗
∗University of Würzburg, Insitute of Computer Science, Würzburg, Germany
<EMAIL_ADDRESS>
+Nokia Networks, Ulm, Germany<EMAIL_ADDRESS>
⋄Tutela Ltd, Victoria, Canada<EMAIL_ADDRESS>Stefan Wunderer+
∗University of Würzburg, Insitute of Computer Science, Würzburg, Germany
<EMAIL_ADDRESS>
+Nokia Networks, Ulm, Germany<EMAIL_ADDRESS>
⋄Tutela Ltd, Victoria, Canada<EMAIL_ADDRESS>Erik Tarnvik⋄
∗University of Würzburg, Insitute of Computer Science, Würzburg, Germany
<EMAIL_ADDRESS>
+Nokia Networks, Ulm, Germany<EMAIL_ADDRESS>
⋄Tutela Ltd, Victoria, Canada<EMAIL_ADDRESS>Andy Hall⋄ ∗University
of Würzburg, Insitute of Computer Science, Würzburg, Germany
<EMAIL_ADDRESS>
+Nokia Networks, Ulm, Germany<EMAIL_ADDRESS>
⋄Tutela Ltd, Victoria, Canada<EMAIL_ADDRESS>
##  1 Introduction

The collection and analysis of meaningful data has always been the key to
optimizing the performance of communication networks. Previously, operators or
providers have captured and analyzed network data and Quality of Service (QoS)
factors at the network level to assess the network perspective. Using
Crowdsourced Network Measurements (CNMs), it is possible to get insights
beyond the network layer, i.e. on application layer and user level towards
Quality of Experience (QoE).

CNMs allow network operators to think outside the box using the mass of end
user devices for gaining measurement data. It gives a much better holistic
understanding of the impact of network challenges or issues on the quality
experienced by end users. The ultimate goal is to use CNMs combined with other
network and user data to improve QoE for applications, but also for regulatory
entities to identify and predict coverage, network, and quality issues. Thus,
in this paper, we will show how CNM data can be used to evaluate the QoS and
QoE of end users in Germany by evaluating the measured throughput and compare
it among different states and cities. Thereby, we will utilize an existing
crowdsourced measurement dataset of Tutela Ltd., covering more than 2.5
million throughput tests in Germany from January until July 2019.

In our paper we will highlight the benefits of going beyond pure speed-tests
by utilizing crowd data. After discussing related work as well as related
measurement approaches, we will go into detail how Tutela performs its
measurements and how these measurements can be interpreted. After that, we
present some exemplary evaluations and show that CNMs can help to get insights
into the user perceived Mobile Broadband (MBB) experience. Not only the
individual measurements play a role here, but rather the overall statement and
the understanding of the validity of an individual value as an overall
statement is in focus, which results in a differentiated view of the
measurements at different network locations and regions. Thus, the results of
the paper are covering detailed throughput statistics for Germany as well as
results for the 16 German states in comparison of some highly as well as
sparsely populated areas. We will show that, for example, Hamburg leads the
ranking of the average download throughput per state with about 18.1 Mbps,
while Saarland is at the low end, having an average of 12.3 Mbps. The paper
shows that the mean throughput as well as the variance differs between
regions. In addition, we map the results to QoE related performance indicators
to include the user perspective of the network.

The remainder of this work is structured as follows. Section 2 provides
background and discusses related work as well as available network reports.
The methodology to measure the network speed is described in Section 3.
Section 4 deals with the measurement data and compares the results of the
German states as well as of different German cities. Finally, Section 5
concludes the paper.

##  2 Background on Crowdsourcing and Related Work

Crowdsourcing is the methodology of processing a task by a large group of
people instead of a designated agent [1]. Best practices and recommendations
for crowdsourced QoE assessment are discovered in [2, 3]. For network
measurements, crowdsourcing has three major advantages: it makes it much
easier to cover a wide range of situations and users, it allows entities other
than the network operator to assess the performance and other characteristics
of a network, independently, with a coverage that is not feasible using other
methods such as drive testing, and it offers the possibility to collect
statistics from end user perspective. A comparison of crowdsourcing with
traditional measurement techniques and best practices how to design
crowdsourced network measurements issues was done in [4]. There are two ways
of doing crowdsourcing studies: either workers are paid to process a task or
applications on the end user’s smartphone are used to collect key performance
indicators (KPIs). With the second way, crowdsourced network measurements can
be seen as a special case of crowdsensing where user devices act as
environmental sensors. Crowdsourced measurement data (crowd data) offers new
possibilities and can be used for various applications such as the
benchmarking of network operators, providers, technologies, or countries as
well as e.g., be used for monitoring, planning, and optimization of the
network. The ultimate goal is to use crowd data – combined with other network
and user data – to improve QoE, but also for regulatory purposes, e.g. to
identify issues with coverage, or network.

In recent years, crowdsourced network measurements become increasingly
relevant in research. One example of this is drive testing. While drive
testing is a commonly used tool to characterize mobile networks, there is
further development towards using a large community and war-driving to
discover WiFi access points and draw a map for coverage [5]. There are a
number of algorithms available for crowdsourced measurement-based cell tower
localization to create a coverage map, e.g., [6, 7]. The coverage map accuracy
from the perspective of device diversity is investigated in [8]. Furthermore,
there are also other ways of measuring and comparing Internet service
providers (ISPs) using crowdsourced data. In [9], the authors used crowd data
collected from peer-to-peer BitTorrent users to compare the performance of
ISPs from end user perspective. Crowdsourced network measurements can also be
done by collecting information during the use of smartphone applications.
Here, different KPIs can be measured on several layers, from context
parameters like cultural background through network parameters like signal
strength up to application parameters like number of stalling and user-focused
parameters like browser session time. Especially video streaming applications
are well used options to collect crowd data on the smartphone of the end users
[10, 11, 12, 13].

Due to this development, the number of crowdsourced network measurement
service providers increased in the last few years, for example,
Tutela111Tutela Technologies, https://www.tutela.com/, Ookla222Ookla,
https://www.speedtest.net/, P3333P3 communications,
http://p3-networkanalytics.com/, QoSi444QoSi, https://www.5gmark.com, and
Opensignal555Opensignal, https://www.opensignal.com/, which use the smartphone
of the end users as measurement device. These companies regularly publish
reports on the mobile network experience in Germany [14, 15, 16, 17], in which
they compare network operators, coverage, and speed of their networks.

Nevertheless, to the best of our knowledge, there is as yet no comprehensive
evaluation of the broadband coverage and connection speed in Germany,
comparing the performance of states and cities on a national level.

##  3 Methodology

The dataset used belongs to Tutela Ltd., an independent crowdsourced data
company with a global panel of over 300 million smartphone users. Tutela
collects data and conducts network tests through software embedded in a
variety of over 3 000 consumer applications. Although started at random times,
measurements are performed in the background in regular intervals if the user
is inactive, and information about the status of the device and the activity
of the network and the operating system are collected. The data is correlated,
grouped and evaluated according to device and network status (power saving
mode, 2G/3G/4G connectivity). Tests are conducted against the same content
delivery network.

Tutela measures the network quality based on the real performance of the
actual network user, including situations when a network is congested, or the
user is throttled by tariff. The results in this paper are based on throughput
testing in which 2 MB files are downloaded via Hypertext Transfer Protocol
Secure (HTTPS). The size reflects the median of the web page size on the
Internet.

###  3.1 Dataset

Within the used dataset for Germany, 3 891 839 crowdsourced network
measurements are included. After filtering incomplete entries, 2 555 557
measurements remain. Data was collected during half a year from January 2019
to July 2019. In addition to meta information like date, geo-coordinates,
region and city, the dataset includes information on current network
performance, including upload throughput, download throughput, average
latency, average jitter, and percentage of packet loss, as well as details
about the used network connection (such as technology and frequency band) as
well as the device (such as OS and device make and model) are collected.

Figure 1: Distribution of all measurements in Germany.

Figure 1 shows the distribution of the measurements on the map. As the data is
collected using crowdsourcing, measuring points are not distributed evenly,
but are often to be seen in urban centers and cities. The mean number of
measurements per square kilometer is about 7 150. Having a look at the
distances between the measurements, the mean distance to the nearest neighbor
is 26.2 m with a maximum of 9 941.89 m and a standard variation of 72.9 m.

Figure 2: Number of measurements per state.

Figure 2 shows the distribution of the number of measurements over the 16
German states. The highest number of measurements can be seen in North Rhine-
Westphalia (759 465 measurements) while the lowest number was measured in
Mecklenburg-Vorpommern (31 552 measurements). Due to the fact that we used
crowdsourced measurements, a high linear correlation (Pearson correlation:
$\tau=0.94$, $p{\text{-}}value\leq 0.01$) between the number of data points
per state and the number of inhabitants is visible.

Figure 3: Number of measurements in Berlin, Hamburg, Munich, Würzburg,
Hildesheim, Cottbus, and Gütersloh.

The number of measurements per cities can exemplarily be seen in Figure 3. In
Germany there are 81 cities with more than 100 000 inhabitants, which are
referred to as so-called large cities (Großstädte). We looked at the cities
individually in the results. In Figure 3, we selected seven German cities
according to their population: Berlin (3 644 826 inhabitants), Hamburg (1 841
179 inhabitants), and Munich (1 471 508 inhabitants) are the three largest
cities, while Hildesheim (101 990 inhabitants), Cottbus (100 219 inhabitants),
and Gütersloh, (100 194 inhabitants) are the smallest ones. In addition, we
included the medium-sized town Würzburg (127 880 inhabitants) to our
evaluations. Thus, it can be seen that even for the smallest city Gütersloh 4
391 measurements were collected.

##  4 Evaluation

To analyze the mobile Internet experience from end user perspective, we
evaluated the collected data from three different aspects. First, we focus on
all measurements and make statements for Germany in general. Afterwards, we
compare the network performance of the 16 different German states. Finally, we
focus on selected German cities and correlate the results with geographic
information.

###  4.1 General Figures for Germany

Table 1: Overall network statistics of Germany. | # measurements | mean | std | min | max | 25% quantile | 50% quantile | 75% quantile
---|---|---|---|---|---|---|---|---
dl throughput | 2 555 557 | 15.3 | 12.9 | 0 | 167.1 | 6.5 | 11.4 | 20.1
ul throughput | 2 555 557 | 7.5 | 6.8 | 0 | 99.9 | 2.4 | 5.1 | 10.8
latency | 2 368 764 | 27.4 | 44.3 | 0.7 | 4862.8 | 16.9 | 20.6 | 26.6
jitter | 2 370 697 | 3.9 | 5.0 | 0 | 199.8 | 1.6 | 2.7 | 4.6

Before comparing the network performance of specific regions or states in
Germany, we will give an overview of the overall network statistics of Germany
based on our dataset. Thus, Table 1 shows basic statistics concerning the
download and upload throughput, latency, and jitter. Due to data privacy
reasons, we will refrain from naming the individual providers or publishing
statistics for each provider. Focusing on the measured download throughput, in
Germany, a mean of 15.4 Mbps with a standard deviation of 12.9 Mbps and a
maximum at 167.1 Mbps was measured. The standard deviation is very high,
resulting in a coefficient of variation of 83.9%, which means a great level of
dispersion around the mean. The mean is therefore a characteristic with poor
informative value. Hence, we go further into detail into the quantiles in
order to better evaluate the throughput numbers. The interquartile range is
6.5 Mbps up to 20.1 Mbps, which means that 75% of all measurement values lie
in this range. 1.3% of the values are below 1 Mbps while 12.8% of the values
are above 30 Mbps.

When looking at the measured download throughput values, the question arises
which apps were able to be served with high QoE. According to [18], the
required throughput is 1 Mbps for the web-based social media platform
Facebook, while 4 Mbps are required for Google Maps. Having a look at video
streaming, Netflix, for example, provides the following recommendations for
video streaming to avoid stalling: 5 Mbps for HD and 25 Mbps for Ultra HD
videos666Internet connection speed recommendation for different video
resolutions: https://help.netflix.com/en/node/306 (Accessed: 2020-01-22).
Concerning social media apps like Facebook, only in 1.3% the download
throughput measurements was lower than 1 Mbps. In 12.6% of our measurements
and thus for every eighth end user, the measured download throughput was lower
than 4 Mbps which probably resulted in bad QoE for using popular apps like
Google Maps. Concerning video streaming, only 17.5% of all users would have
been able to stream Ultra HD videos with high satisfaction.

When looking at the other measured values like upload throughput, latency, and
jitter, it can also be seen that the standard deviations are very high, too.
For the upload throughput, the mean value is 7.5 Mbps with a standard
deviation of 6.8 Mbps, which means a coefficient of variation of 89.7%.

Figure 4: CDF of the download throughput per access technology, cut off at 100
Mbps.

Having a look at the used access technology, in 66.8% 4G was used, in 32.9%
3G, in less than 0.1% 2G, and in 0.3% the technology could not be identified.
We assume that the 2G value should be used with caution due to a certain
measurement bias. The measurement method requires smartphones, which means
that the proportion of 2G throughput measurements does not correspond to the
actual proportion of 2G users in the network (consider the mass of 2G IoT
devices), but refers to smartphones with a 2G connection. Figure 4 shows the
cumulative distribution function (CDF) of the download throughput per access
technology as well as for all measurements, cut off in the figure at 100 Mbps.
For 3G, a mean download throughput of 7.7 Mbps with a maximum of 77.9 Mbps and
a standard deviation of 4.1 Mbps was measured. In comparison, for measurements
using 4G, a mean of 19.1 Mbps with a maximum of 167.1 Mbps and a standard
deviation of 14.0 Mbps was recorded. While the mean value is higher here, the
coefficient of variation is also significantly higher (73.4%) compared to 3G
(53.3%).

By comparing the values with the required specifications in the IMT-2000 [19]
for 3G and IMT-Advanced [20] for LTE, the values for 3G are on average above
the required throughput values of 2 Mbps for stationary users. The required
values for pedestrians and moving vehicles are 384 kbps and 144 kbps for
vehicles in IMT-2000. For 4G, the values are significantly below the peak
rates of the 3GPP LTE or LTE Advanced specifications with 327 Mbps,
respectively 1000 Mbps with low mobility. The IMT-Advanced requires a
throughput rate in the downlink and uplink of 1 Gbps. However, if you
calculate an LTE performance with 5 MHz, 64-QAM, an error correction rate of
5/6 and frame duration of 0.5 ms (200 frames per second), you get 19 Mbps or
about 38 Mbps for 2x2 MIMO. Taking into account that the German mobile
providers usually try to deploy 10 MHz or larger bands, the results show that
other factors such as channel quality to the user or the number of users in a
cell have a significant influence on the actual throughput rates.

###  4.2 German State Comparison

Figure 5: Download throughput per region.

To go more into detail, we investigated the differences between the network
conditions in the 16 German states. Figure 5 shows boxplots of the measured
download throughput per state. The median is marked in each box as black
vertical, while the mean is presented as white rectangle. 6 out of 16 states
have a mean download throughput higher than 16 Mbps. The highest mean download
throughput can be seen in Hamburg (18.1 Mbps) and Berlin (17.9 Mbps), lowest
in Saarland (12.3 Mbps). Nevertheless, Hamburg has the second highest standard
deviation of 14.8 Mbps (headed by Berlin with 14.9 Mbps), while Saarland with
10.3 Mbps has the lowest standard deviation. Thus, it is shown that the mean
download throughput differs significantly among all 16 states.

Next, we focus on the upload throughput, where a similar picture can be drawn.
The means of the upload throughput per state show significant differences.
Here, again Hamburg (8.2 Mbps) and Berlin (8.2 Mbps) show the highest values
while Saarland (6.6 Mbps) the lowest.

Looking at the correlation between the measured network characteristics and
statistics of each state, influencing factors can be found. For the mean
download throughput and the population density per state, a significant
positive linear relationship (Pearson correlation: $\tau=0.57$,
$p{\text{-}}value=0.02$) is visible. Thus, the higher the number of
inhabitants per km2, the higher the mean download throughput, which shows the
deployment strategies of German mobile network providers, probably due to
economic reasons. In addition, we found that the standard deviation of the
download throughput per region increases significantly for increasing
population density (Pearson correlation: $\tau=0.51$,
$p{\text{-}}value=0.04$). For example, Berlin, with a population density of 4
090 inhabitants per km2, has a standard deviation of 14.9 Mbps for the
download throughput, while Saarland (385 inhabitants per km2) has a standard
deviation of 10.3 Mbps for the download throughput. Furthermore, a significant
correlation (Pearson correlation: $\tau=0.59$, $p{\text{-}}value=0.01$)
concerning the mean upload throughput and the population density was found.
Focusing on the influence factors of jitter, a correlation to the size of the
state (Pearson correlation: $\tau=0.59$, $p{\text{-}}value=0.02$) as well as
the population (Pearson correlation: $\tau=0.52$, $p{\text{-}}value=0.04$) is
visible.

To investigate the influence of the used access technologies in each state,
Table 2 shows the percentage of measurements conducted using 4G, 3G, 2G, or
unknown technology. Again, we assume that the value for 2G should be used with
caution due to the measurement methodology, see Section 4.1. Berlin, Hamburg,
and Bremen show the highest number of 4G measurements, while Brandenburg,
Thuringia, and Saarland the lowest. In all states, at most 0.01% of the
measurements were done using a smartphone with 2G access. By correlating the
mean download as well as the mean upload throughput with the percentage of 4G
measurements, high positive linear correlations are visible (Pearson
correlation download: $\tau=0.80$, $p{\text{-}}value<0.01$, upload:
$\tau=0.79$, $p{\text{-}}value<0.01$). This result matches our previous
findings that on average measurements conducted using 4G have a significant
higher download and upload throughput than using 3G. Concerning jitter and
latency, the used technology showed no influence.

Table 2: Usage of access technologies in the dataset | 4G | 3G | 2G | unk.
---|---|---|---|---
Baden-Württemberg | 62.2 | 37.4 |  $<$0.1 | 0.3
Bavaria | 67.6 | 32.1 |  $<$0.1 | 0.3
Berlin | 74.8 | 24.8 |  $<$0.1 | 0.4
Brandenburg | 61.6 | 38.2 |  $<$0.1 | 0.2
Bremen | 69.2 | 30.7 |  $<$0.1 | 0.1
Hamburg | 74.5 | 25.2 |  $<$0.1 | 0.3
Hesse | 65.4 | 34.2 |  $<$0.1 | 0.4
Lower Saxony | 66.8 | 32.9 |  $<$0.1 | 0.4
Meck.-Vorpom. | 63.0 | 36.8 |  $<$0.1 | 0.2
North Rhine-Westph. | 66.9 | 32.7 |  $<$0.0 | 0.3
Rhineland-Palatinate | 63.6 | 36.1 |  $<$0.1 | 0.3
Saarland | 59.0 | 40.6 |  $<$0.1 | 0.3
Saxony | 68.2 | 31.3 |  $<$0.1 | 0.5
Saxony-Anhalt | 66.2 | 33.4 |  $<$0.1 | 0.4
Schleswig-Holstein | 68.7 | 31.1 |  $<$0.1 | 0.3
Thuringia | 60.9 | 38.6 |  $<$0.1 | 0.4

###  4.3 German City Comparison

Figure 6: Download throughput per city for Berlin, Hamburg, Munich, Würzburg,
Hildesheim, Cottbus, and Gütersloh.

Finally, we will have a look at the measurement results broken down to
individual cities. Starting again with the download throughput, Figure 6 shows
the cumulative distribution of the measured values for seven selected cities.
Here, Hildesheim and Hamburg show the highest mean download throughput with
18.2 Mbps and 18.1 Mbps, while Gütersloh and Würzburg show the lowest means
(13.7 Mbps and 12.5 Mbps). Nevertheless, Gütersloh and Würzburg show the
lowest standard deviations (11.1 Mbps and 10.9 Mbps), while Berlin and Hamburg
show the highest (14.9 Mbps and 14.8 Mbps). In general, cities with a high
mean throughput must naturally also have a higher variation in values.

Having a look at all 81 German large cities (Großstädte), we measured the
highest mean download throughput in Potsdam (21.3 Mbps) and Braunschweig (20.1
Mbps) and the lowest in Reutlingen (10.8 Mbps) and Bremerhaven (10.9 Mbps).
Having a look at the standard deviation, Saarbrücken (9.9 Mbps) and Reutlingen
(10.1 Mbps) showed the lowest values while the highest variations were found
in Dresden (15.8 Mbps) and Potsdam (15.4 Mbps).

Putting the download throughput in relation to the size of the city, a weak
linear correlation (Pearson correlation: $\tau=0.33$ $p{\text{-}}value\leq
0.01$) is visible. Furthermore, also the standard deviation of the download
throughput is influenced by the size of the city (Pearson correlation:
$\tau=0.40$ $p{\text{-}}value\leq 0.01$) as well as by the number of
inhabitants (Pearson correlation: $\tau=0.32$ $p{\text{-}}value\leq 0.01$).
This correlation could be caused by the fact that larger cities have a high
number of inhabitants and thus, the economic benefit for network operators is
higher and therefore provides an economical incentive to upgrade the network.
This result is supported by the finding that number of inhabitants as well as
the size of the city shows a significant linear correlation to the percentage
of 4G measurements (Pearson correlation population: $\tau=0.34$
$p{\text{-}}value\leq 0.01$, size: $\tau=0.33$ $p{\text{-}}value\leq 0.01$).

In general, for all of the 81 large cities, the mean download and upload
throughput increases by increasing percentage of 4G measurements. Here, a
significant correlation of $\tau=0.64$ ($p{\text{-}}value\leq 0.01$) for
download, and a correlation of $\tau=0.69$ ($p{\text{-}}value\leq 0.01$) for
upload throughput is visible. In this case, jitter and latency decrease.

##  5 Conclusion

In this paper, the mobile Internet experience for Germany was evaluated with
the help of crowdsourcing from the perspective of an end user. Based on a
dataset from Tutela Ltd. with 2.5 million throughput measurements from January
to July 2019, statistics for various German areas were created and examined.
Insights into this emerging methodology of crowdsourced network measurements
were given and the advantages of this method were highlighted. The paper
contains statistics and conclusions for several large cities and regions in
Germany compared to general statements for Germany. One contribution of the
paper is to show how the measured values differ between individual
measurements and the mean value for an area.

The results present a heterogeneous measurement result. Compared to the
figures throughout Germany, high variances in the measurements are already
visible for the states. This is mainly due to the deployment status of the
network, which is being driven forward in areas with higher population numbers
due to economic factors. Furthermore, higher throughput values are to be seen
in areas with high population density. For Germany, the interquartile range is
6.5 Mbps up to 20.1 Mbps, which means that 75% of all measurement values lie
in this range. 1.3% of the users got less then 1 Mbps throughput which is
below the recommended value to use web-based apps such as Facebook according
to [18]. For every eighth end user, the measured download throughput was lower
than 4 Mbps which probably resulted in bad QoE when using popular apps like
Google Maps. Only 17.5% of all users would have been able to stream Ultra HD
videos with high satisfaction. The 3G throughput for Germany according to our
dataset lies at 7.7 Mbps. The 4G throughput was measured at 19.1 Mbps. While
the mean value is higher here, the coefficient of variation is also
significantly higher (73.4%) compared to 3G (53.3%). Going into detail about
the cities, the differences in the throughput are even more severe. Hamburg
has the highest average throughput with 18.1 Mbps, while Saarland is at the
bottom of the list with 12.3 Mbps. It is shown that the mean download
throughput differs significantly among all 16 states.

Overall, it can be concluded that, firstly, individual views are necessary
and, secondly, different regions have to be considered in order to obtain a
holistic picture of the status of the mobile network performance in Germany.
The throughput is higher in densely populated areas such as large cities than
in smaller cities. Overall, mean downlink throughput and city size correlate,
and standard deviation of downlink throughput and city size correlate. Large
cities are therefore better developed, but also fluctuate more in throughput.

## References

  * [1] J. Howe, “The Rise of Crowdsourcing,” _Wired magazine_ , vol. 14, no. 6, pp. 1–4, 2006.
  * [2] T. Hossfeld, C. Keimel, M. Hirth, B. Gardlo, J. Habigt, K. Diepold, and P. Tran-Gia, “Best Practices for QoE Crowdtesting: QoE Assessment with Crowdsourcing,” _IEEE Transactions on Multimedia_ , vol. 16, no. 2, pp. 541–558, 2013.
  * [3] T. Hoßfeld, M. Hirth, J. Redi, F. Mazza, P. Korshunov, B. Naderi, M. Seufert, B. Gardlo, S. Egger, and C. Keimel, “Best Practices and Recommendations for Crowdsourced QoE-Lessons learned from the Qualinet Task Force "Crowdsourcing",” 2014.
  * [4] M. Hirth, T. Hoßfeld, M. Mellia, C. Schwartz, and F. Lehrieder, “Crowdsourced Network Measurements: Benefits and Best Practices,” _Computer Networks_ , vol. 90, pp. 85–98, 2015.
  * [5] M. Kim, J. J. Fielding, and D. Kotz, “Risks of using AP Locations Discovered through War Driving,” in _International Conference on Pervasive Computing_. Springer, 2006, pp. 67–82.
  * [6] E. Neidhardt, A. Uzun, U. Bareth, and A. Küpper, “Estimating Locations and Coverage Areas of Mobile Network Cells based on Crowdsourced Data,” in _6th Joint IFIP Wireless and Mobile Networking Conference (WMNC)_. IEEE, 2013, pp. 1–8.
  * [7] Z. Li, A. Nika, X. Zhang, Y. Zhu, Y. Yao, B. Y. Zhao, and H. Zheng, “Identifying Value in Crowdsourced Wireless Signal Measurements,” in _Proceedings of the 26th International Conference on World Wide Web_ , 2017, pp. 607–616.
  * [8] M.-R. Fida and M. K. Marina, “Impact of Device Diversity on Crowdsourced Mobile Coverage Maps,” in _2018 14th International Conference on Network and Service Management (CNSM)_. IEEE, 2018, pp. 348–352.
  * [9] Z. S. Bischof, J. S. Otto, M. A. Sánchez, J. P. Rula, D. R. Choffnes, and F. E. Bustamante, “Crowdsourcing ISP Characterization to the Network Edge,” in _Proceedings of the First ACM SIGCOMM Workshop on Measurements Up the Stack_. New York, NY, USA: ACM, 2011, pp. 61–66.
  * [10] F. Wamser, M. Seufert, P. Casas, R. Irmer, P. Tran-Gia, and R. Schatz, “YoMoApp: A tool for Analyzing QoE of YouTube HTTP Adaptive Streaming in Mobile Networks,” in _2015 European Conference on Networks and Communications (EuCNC)_. IEEE, 2015, pp. 239–243.
  * [11] S. Wassermann, P. Casas, M. Seufert, and F. Wamser, “On the Analysis of YouTube QoE in Cellular Networks through in-Smartphone Measurements,” in _12th IFIP Wireless and Mobile Networking Conference (WMNC)_ , Paris, France, Sep. 2019. [Online]. Available: https://hal.archives-ouvertes.fr/hal-02159716
  * [12] A. Schwind, C. Midoglu, Ö. Alay, C. Griwodz, and F. Wamser, “Dissecting the Performance of YouTube Video Vtreaming in Mobile Networks,” _International Journal of Network Management_ , p. e2058, 2019.
  * [13] A. Schwind, F. Wamser, S. Wunderer, C. Gassner, and T. Hoßfeld, “Mobile Internet Experience: Urban vs. Rural – Saturation vs. Starving?” _arXiv preprint arXiv:1909.07617_ , 2019.
  * [14] 4Gmark, “Data Mobile Barometer,” February 2018. [Online]. Available: https://www.5gmark.com/news/2017/4Gmark-Barometer-2017-DE.pdf
  * [15] Tutela Technologies, “Germany Mobile Experience Report – Country-level mobile experience and usage results from Tutela’s crowdsourced mobile network testing (May-July 2019),” July 2019. [Online]. Available: https://insights.tutela.com/report/germany-mobile-experience-report-july-2019/view/
  * [16] P3 Communications, “The Great 2019 Mobile Network Test,” January 2019. [Online]. Available: http://p3-networkanalytics.com/wp-content/uploads/2018/12/Network-Test-2019-connect-2019-01-English_sp.pdf
  * [17] Opensignal, “Germany – Erfahrungsbericht mit mobilem Netzwerk (November 2019),” November 2019. [Online]. Available: https://www.opensignal.com/de/reports/2019/11/germany/mobile-network-experience
  * [18] P. Casas, R. Schatz, F. Wamser, M. Seufert, and R. Irmer, “Exploring QoE in Cellular Networks: How much Bandwidth do you need for popular Smartphone Apps?” in _Proceedings of the 5th Workshop on All Things Cellular: Operations, Applications and Challenges_ , 2015, pp. 13–18.
  * [19] ITU-R, “International Mobile Telecommunication-200 (IMT-2000),” International Telecommunication Union, Recommendation M.687-2, 2019.
  * [20] ——, “International Mobile Telecommunication-Advanced (IMT-Advanced),” International Telecommunication Union, Recommendation M.2012-4, 2019.