**affiliationtext<EMAIL_ADDRESS>Department of Industrial
and Systems Engineering, North Carolina State University, Raleigh,
NC22affiliationtext: Operations Research Program, North Carolina State
University, Raleigh, NC

# Electric Vehicles Limit Equitable Access to Essential Services During
Blackouts

Yamil Essus  Benjamin Rachunok

###### Abstract

Electric vehicles (EVs) link mobility and electric power availability, posing
a risk of making transportation unavailable during blackouts. We develop a
computational framework to quantify the impact of EVs on mobility and access
to services and find that existing access issues are exacerbated by EVs. Our
results demonstrate that larger batteries reduce mobility constraints but
their effectiveness is dependent on the geographic distribution of services
and households. We explore the trade-offs between mobility and quality-of-life
improvements presented by Vehicle-to-Grid technologies and the feasibility and
trade-offs of public charging infrastructure as a solution to access
inequalities. Equitable access to essential services (e.g. supermarkets,
schools, parks, etc.) is the most important aspect of community resilience and
our results show vehicle electrification can hinder access to essential
services unless properly incorporated into policy and city-scale decision-
making.

Vehicle electrification is a key component of Sustainable Development Goals
yet the mass adoption of electric vehicles can lead to unintended consequences
on community mobility during natural hazards. A growing body of research has
identified positive and negative unintended consequences of the large-scale
transition from gas to battery electric vehicles (BEVs). Beyond
decarbonization, BEVs have the potential to positively impact air quality and
environmental justice [1, 2, 3, 4, 5, 6] and to provide quality-of-life
improvements through Vehicle-to-grid (V2G) technologies as vehicles can be
used to power homes during outages [7, 8]. However, there are growing
questions about BEV carbon efficiency [9, 10] and mass-production feasibility
[11, 12, 13, 14, 15] and BEVs also introduce challenges for integration into
the grid[16], create infrastructure vulnerabilities [17] and ties between
efficiency and consumer behavior [18].

BEVs also pose a challenge to owners during natural disasters [19, 20].
Electric vehicles couple transportation and electric power systems and this
link poses a risk of power outages making transportation unavailable during
disasters and limiting mobility for BEV owners during power outages. Because
electric vehicles require periodic charging, the mobility potential of
individual households is limited by their access to electricity and prolonged
blackouts will constrain the ability of car-dependent households to access
essential services.

Due to the direct relation between driving distance to services and vehicle
battery consumption, mobility will be impacted by geographic and technological
factors. Geography determines the driving distance to essential services which
will be translated to electricity consumption devoted to transportation.
Technology determines the size and efficiency of electric vehicle batteries
[21, 22] while V2G technologies introduce tradeoffs between quality of life
improvements from powering household amenities and using electricity for
mobility.

The linkage between mobility, power availability and quality-of-life has broad
implications for community resilience. Beyond the traditional focus on
infrastructure functionality, equitable access to essential services is the
most important aspect of community resilience [23]. In 2021, the
transportation sector accounted for roughly 22% of CO2 emissions [24, 25] and
55% of transportation emissions in the US are attributed to gasoline vehicles
[26]. Accordingly, battery electric vehicles are at the forefront of
Sustainable Development Goals [27, 28, 29, 30, 31] and the mass adoption of
BEVs must be carefully managed to avoid negative impacts on community
resilience.

We develop a computational modeling framework to quantify the impact of
vehicle electrification on limited mobility and access to essential services
in urban areas during prolonged blackouts. We define measures of access risk
and evaluate how risk changes across large urban centers in the U.S. We find
inequalities in access to essential services will be exacerbated by vehicle
electrification during blackouts. We also find that urban areas with high
population density are associated with lower levels of access risk whereas
high car ownership rates correlate with higher access risk. Moreover, we test
different electric vehicle technologies and find that increased battery size
lowers access risk by increasing potential driving distance, however the
impact of battery size is highly dependent on the geography of each city.
Finally, we test how V2G creates a trade-off between access to services and
use of household amenities and find that V2G disproportionately benefits
access-rich households.

### The link between mobility and electric power availability

We model the impact of electrification on access to essential services and how
access is mediated by technology and geography. We measure access risk at a
household level for 16 major cities in the US. For each census block group in
the city, we calculate the average distance to the nearest supermarket,
school, and park and assume each household makes a given number of trips to
each service per week (Methods). Driving distance is converted to weekly
electricity consumption using standard BEV battery consumption rates [32] and
we assume a fully charged battery at the start of the blackout to model
available energy as a function of time. We define a measure of access risk
which considers households at risk if two weeks of mobility activities
consumes more than 50% of their vehicle’s energy reserves and assume a battery
capacity of 60 kWh which is roughly equivalent to a Tesla Model Y (2022) and
around the average capacity of available vehicles [32]. Starting with a full
battery, the charge remaining after a two-week blackout represents the slack
in mobility a household has after meeting basic mobility needs. We chose a
two-week blackout because it lies in the range of recent events [33, 34] and
it allows us to explore the interactions between access risk and electric
vehicles.

We capture geographic inequality by computing driving distance to essential
services including schools, supermarkets and parks. Figure 1.a shows driving
distance to the closest school for each census block group in Charlotte, NC
and Figure 1.b shows the distribution of driving distance for all services.
Figure 1.c shows the distribution of the State of Charge (SOC) over time. By
evaluating the distribution of SOC we are able to identify census block groups
at higher risk of losing access to essential services (households which are
access-poor) as well as those with relatively little impacts (households which
are access-rich). For example, it takes nine days for the bottom 10% of the
population to become at risk of losing access, whereas the top 10% could
withstand over 60 days without power before becoming at risk. By integrating
characteristics of electric vehicles and driving distances at a census block
group level we are able to assess the equity across block groups in terms of
remaining battery resulting from existing inequalities in driving distance to
services.

Figure 1: The effect of geographic inequalities on access to essential services. (a) Driving distance to the closest school in Charlotte, NC. (b) distribution of distance to closest services for the combined statistical area of Charlotte, NC. and (c) Evolution of remaining battery available for 10th-90th percentile levels. Map generated using TIGER files from the US Census Bureau, downloaded in April 2023.  Name | Mean [75th - 25th] Remaining Battery | Average Car Ownership | Average Density
---|---|---|---
New York, NY | 51.26 [56.10 - 49.78] | 0.52 | 12769
Salt Lake City, UT | 50.20 [53.99 - 49.74] | 0.79 | 1987
Los Angeles, CA | 50.07 [54.34 - 49.25] | 0.74 | 3940
Chicago, IL | 49.71 [54.91 - 47.85] | 0.70 | 3354
Miami, FL | 47.85 [53.06 - 45.17] | 0.73 | 3007
Portland, OR | 47.33 [54.10 - 47.59] | 0.77 | 1945
Dallas-Fort Worth, TX | 45.72 [52.46 - 44.54] | 0.77 | 1680
Kansas City, MO | 44.54 [52.44 - 41.89] | 0.80 | 983
Houston, TX | 44.30 [51.50 - 42.47] | 0.75 | 1835
Orlando, FL | 41.41 [48.97 - 37.12] | 0.75 | 1317
Cincinnati, OH | 40.78 [50.17 - 34.27] | 0.79 | 1001
St. Louis, LA | 39.14 [49.40 - 32.80] | 0.80 | 868
Atlanta, GA | 38.74 [46.75 - 32.99] | 0.77 | 1039
Nashville, TN | 36.41 [47.75 - 26.19] | 0.81 | 763
Raleigh, NC | 36.11 [48.72 - 30.62] | 0.79 | 788
Charlotte, NC | 35.70 [46.65 - 28.62] | 0.79 | 728
Table 1: Simulation Results. For each Combined Statistical Area we include
average remaining battery [kWh] as well as population weighted 25th and 75th
percentile. Additionally, average population density [people/km2] and car
ownership rates [cars/person on household] were obtained from open census
data.

### Inequalities in access to essential services will be exacerbated by
electric vehicles during blackouts

We find that existing inequalities in geographical distribution of access to
services will be exacerbated by the electrification of private cars because
households further away from amenities will use more of their available energy
on each trip. Accordingly, households in access-poor areas will be at risk of
losing mobility due to this link between transportation and electricity.
Average, 25th and 75th percentile values for the remaining battery of each
city can be found in Table 1. In the 16 cities tested, we find that more
equitable access to essential services is associated with lower access risk.
We used the Equally Distributed Equivalent (EDE) Index to quantify access
inequalities. The EDE has been used to study inequality in urban settings with
a focus on equitable distribution of burdens and resources [35] and can be
interpreted as the value that would make an individual indifferent between the
current distribution and complete equality at the level of the EDE. The EDE
considers both the average value and the dispersion of a distribution, so that
the EDE of two cities can be directly compared. Moreover, if two distributions
share the same average, the EDE will penalize the one with higher
dispersion.For example, the EDE of remaining battery of Portland is 43.6 kWh,
which would increase if either the average remaining battery in Portland
increases or if the inequality is reduced. The maximum EDE of remaining
battery in Portland is limited to 60kWh and is achieved when every household
has exactly 60kWh remaining. Details about the EDE metric used are included in
the Methods section.

We find a strong correlation between inequality in battery levels and access
risk in each city we tested (Figure 2.a). For example, Chicago has an EDE of
47.1 kWh and only 5% of the residents would be at risk of becoming
disconnected from essential services while Nashville has a EDE of 28.4 kWh and
30% of residents experience access risk. Differences in geographic access
between access-poor and access-rich households inside a city result in battery
inequality that is captured by the EDE. The trend of greater EDE being
associated with lower access risk demonstrates the link between the geographic
aspects that drive consumption rates and our measure of risk as well as how
BEVs exacerbate existing geographic inequalities during prolonged blackouts.

Access issues also demonstrate spatial trends. This is evidenced visually in
the map in Figure 2.b and the population distribution in Figure 2.c colored
according to the remaining battery of each census block after the blackout. We
categorize each census block group according to estimated remaining energy
using the intervals 0-10 kWh (Very Low), 10-20 kWh (Low), 20-40 kWh (Medium),
40-50 kWh (High), 50-60 kWh (Very High). Low remaining battery is associated
with high risk, and high remaining battery with low risk. The same color
palette was used for the maps and the population bar plots. In all three
cities we identify regions of good access with short driving distances to
services and sparse areas with poor connectivity that result in low battery
for households located there. For example, Nashville has one center of very
high remaining battery surrounded by large areas of medium to low remaining
battery. On the other hand, Houston has a large number of small regions of
high connectivity that end abruptly when moving away from the city center.
Lastly, Chicago has a large area with very high remaining battery and only
peripheral areas show low or very low levels. These results indicate that the
geographic distribution of services and households greatly affects the access
risk levels of an urban area.

We note that despite the fact geographic inequalities drive access risk, there
is a significant number of people experiencing access risk in cities with
relatively low inequality (Figure 2b). For example, both inequality and access
risk are high in Nashville, TN, whereas Chicago, IL shows low inequality and
low access risk, with over 80% of the total population in the High or Very
High level of battery remaining. Nonetheless, there is still over 500,000
people in Chicago experiencing access risk. These results indicate that the
absolute and relative size of the population experiencing access risk are
important for assessing how EVs effect mobility.

Figure 2: Geographic inequalities linked to access risk (a) Equally
Distributed Equivalent against percentage of residents experiencing access
risk for Combined Statistical Areas in the U.S. (b) Available energy (kWh)
after a two week blackout as a function of distance to amenities. (c)
Population of the three cities shown in the map colored according to remaining
battery levels. Maps were generated using TIGER files from the US Census
Bureau database Redistricting 2022, downloaded in April 2023

### High population density and low car ownership rates are correlated with
lower access risk

We find geographic inequality and access risk are associated with urban growth
patterns. We quantify urban growth using population density and car ownership
rates (Figures 3.a and 3.b) following research that has linked these metrics
to social inequality in the context of sustainable development. For instance,
inequalities in urban population density within a city can appear as a result
of socio-political drivers, produce economic inequalities and propagate
patterns of segregation [36]. Additionally, the dependence on cars of American
households [37, 38] is an obstacle for the development of sustainable cities
because of its ties to income inequality [39].

Population density is negatively correlated with access risk and car ownership
rates are positively correlated with access risk across the U.S. Population
density is inversely proportional to travel distance in the U.S. and is
largely reflective of other dimensions of disadvantage such as economic
inequality [40]. Additionally, car ownership rates quantify the perceived need
for a personal car and the lack of reasonable public transport alternatives
[39]. Figure 3.b shows the aggregated value of density and car ownership
variables for each city and the trend between them and access risk. Across
cities we tested, we find that low population density and high car ownership
correlate with high risk. These trends show the interaction between urban
growth metrics and access risk and are evidence of how urban planning that
incentivizes high population density and focuses on transportation policies
that disincentivize car ownership can lead to lower access risk.

We also find that the trends between density, car ownership, and access risk
are present within cities. Figure 3.a shows population density, average
remaining battery and average car ownership rates for each census tract in Los
Angeles, CA and Atlanta, GA. For example, the top and bottom 25% most densely
populated tracts in Atlanta have an average remaining battery of 45kWh [75%
battery level] and 23kWh [38% battery level] respectively, while top and
bottom 25% by car ownership rates have 33kWh [55% battery level] and 42kWh
[70% battery level] respectively, all against a city average of 37.5kWh [63%
battery level]. For Los Angeles, the top and bottom 25% most densely populated
tracts have an average remaining battery of 52kWh [87%] and 40.5kWh [68%]
respectively, while top and bottom 25% by car ownership rates have 45kWh [75%]
and 51kWh [85%] respectively, all against a city average of 48.5kWh [82%].
Associations between population density and decreased access risk as well as
car ownership and increased access risk highlight the importance of
identifying vulnerable populations even when the average risk for the entire
area is relatively low.

Figure 3: Urban form and access risk (a) Log-minmax-scaled population density,
battery SOC after a two week blackout and minmax-scaled car ownership rate for
each census block group. (b) Scaled population density and scaled car
ownership plotted against percentage of population experiencing access risk
for each of the CSA considered in the study. Values were aggregated from block
group level using population-weighted average. New York City was excluded from
this plot because it was considered an outlier. Maps were generated using
TIGER files from the US Census Bureau database Redistricting 2022, downloaded
in April 2023.

### The effectiveness of technological improvements on access risk depends on
geography

We find increased battery capacity or vehicle efficiency reduces access risk,
but some areas are more sensitive to technological changes than others. We
model improved vehicle energy efficiency or larger batteries as having the
same effect. Greater efficiency reduces consumption per BEV trip and bigger
batteries allow households to make more trips. We find that for all cities,
technological upgrades reduce access risk but the magnitude of the decrease
depends on the geographical distribution of households and services. We test
the effects of reducing or increasing battery capacity on the portion of
households experiencing access risk in each city. Three scenarios of initial
battery capacity were tested: Low (40 kWh), Medium (60 kWh, base scenario) and
High (80 kWh).

Figure 4.a shows the percentage of residents experiencing access risk for each
scenario in each city. All cities follow the trend of risk being inversely
proportional to battery size but geographic differences impact the change in
access risk. For example, Orlando presents similar levels of access risk as
Kansas City in the high battery scenario, but with low capacity batteries 10%
more of Orlando’s households experience access risk compared to Kansas City’s
low battery scenario. This happens because both cities share a similar portion
of very access-poor households that even in the high battery scenario are at
risk, but there are more households that take advantage of the incremental
increase in battery in Orlando. This means Kansas City’s population is more
concentrated into the extremes of access-rich households that are never at
risk and access-poor households that require a bigger jump in battery to not
experience risk.

Figure 4.b highlights the portion of the households at risk in the low battery
scenario that are not at risk in the high battery scenario for all cities
tested. This value represents the maximum improvement in access risk that a
change from 40 kWh to 80 kWh can achieve. For example, Atlanta’s geography
allows for a decrease of 77% in the percentage of people at risk when the
technological scenario is favorable. On the other hand, Salt Lake City
presents an improvement of around 50% when comparing a Low battery vehicle
scenario to a High battery vehicle scenario. This happens largely because of
the geographical distribution of the population and how technological changes
aligns with it. We can see in the map in Figure 4.c that Atlanta’s population
that benefits from an increased battery capacity gradually increases as we
move away from the urban center. In contrast, Salt Lake City’s map shows that
most people living in the urban center are already in good standing when it
comes to access risk, while people in the outskirts required an even larger
increase in capacity to make it above the risk threshold.

Figure 4: Impacts of battery size on access risk (a) Percentage of residents
experiencing access risk in a low, medium and high technology scenario (40, 60
and 80 kWh of initial battery charge respectively) for each CSA considered in
this study. (b) Percentage of the portion of residents at risk in the low
technology scenario that are not at risk in the high technology scenario for
each CSA and (c) Geographic location of census block groups at risk that
require None/Medium/High battery capacity to evade the risk. Maps were
generated using TIGER files from the US Census Bureau database Redistricting
2022, downloaded in April 2023

### The trade-offs between Vehicle-To-Grid and mobility

Vehicle-to-grid (V2G) is a technology that enables energy to be returned to
the grid or a house from a car battery [41] and can provide significant
quality of life improvements, but introduces a trade-off between mobility and
household amenities. This trade-off is mediated by geography because it is
determined by the rate at which households require electricity to access
essential services. Using average household consumption rates, we explore the
relation between mobility and household amenities for each geographic area.

Households with better access to services will sacrifice more days of mobility
for each day of house amenities. This happens because better access leads to
less energy consumption per trip therefore the same energy consumption on
amenities for an access-rich household represents a larger number of days of
mobility than for an access-poor household. Figure 5 shows the distribution of
this trade-off across the population of each city. We find that inequalities
in access appear as differences in mobility-V2G trade-offs within each city.
For instance, in Miami, FL some households have a trade-off of up to 7 weeks
of mobility per day of V2G while the median household has a trade-off of 3.8
weeks per day of V2G. We also find differences across cities (Figure 5). The
mobility-V2G trade-off experienced by the median household in NYC corresponds
to around 4.5 weeks of mobility per day of V2G. We see that this level of V2G
availability is better than the one experienced by 62% of the population in
Portland, OR and 97% in Charlotte, NC. These results indicate that better
access to services not only results in reduced access risk but also improves
the usefulness of V2G.

Due to the high consumption of electricity for household amenities in
comparison to vehicle batteries and mobility consumption, V2G technologies can
only be realistically used by access-rich households. Estimated daily
consumption in house amenities amounts to 13.3kWh (see Methods). Given the
characteristics of the BEV considered in this study one day of V2G consumption
amounts to weeks of mobility to access essential services. This means that
only the households that are the least likely to experience access risk are
will be likely to use this technology.

Figure 5: Vehicle-to-grid and Mobility Trade-offs. Estimated distribution of
the equivalent weeks of access to services per each day of average V2G
consumption across the population of each city. Percentage labels show the
portion of the population that has a mobility-V2G trade-off of at most 4.5
weeks of mobility per day of V2G, corresponding to the median of New York
City. The figure was trimmed at 8 weeks of mobility per day of V2G. Each
distribution is shown with its own normalized scale in the y-axis while all
share the x-axis scale.

## Discussion

Using a computational framework informed by publicly available data, we show
that access risk is present in urban environments with mass adoption of
electric vehicles. Geographic inequality is translated into unequal
electricity consumption rates which leads to an increased risk of losing
access to essential services. Poor access gets exacerbated resulting in rapid
consumption and low battery levels, thus, the inequality in remaining energy
represents the weight of the lower end of the distribution of this risk for a
given population. The geographic distribution of households and services is
one of the main drivers of access risk. All else remaining equal, changes in
geography will greatly impact access risk on almost all settings we tested.
Our results indicate that policies that impact transportation systems,
population density (such as district zoning laws) or car ownership rates (such
as parking minimums and tax credit incentives) will all impact equitable
disaster outcomes. This highlights how BEV mass adoption creates additional
interdependencies in communities that must be included in policy decision-
making.

A key result is that increases in battery capacity will reduce access risk but
the effectiveness of larger batteries depends on geography. Electric vehicle
technology is evolving quickly and there is rapid improvement in the battery
capacity and efficiency of personal vehicles to overcome range limitations.
These improvements can alleviate the burdens of access risk, but our results
show that geographic disparities produce unequal benefits from increased
battery sizes or efficiency gains. Because of this, technology-based public
policies aiming to reduce access risk (e.g. economic incentives for more
efficient or higher capacity vehicles) will have different outcomes depending
on the geographic distribution of the population and services, and need to be
evaluated for each region individually. For example, the clean vehicle tax
credit [42] subsidies EV purchases with up to $ 7,500 in tax credits. Since
the cost of the battery remains the most important cost of the manufacturing
of BEVs [43, 44], we estimate this translates into 7.5 kWh of additional
battery assuming all the incentive is used towards increasing battery capacity
(see Methods Section). This increase in battery would reduce households
experiencing access risk by 9.2% of total population in Charlotte, but only by
1.4% in New York or 1.6% in Chicago.

Another potential solution to EV-induced access risk is public charging.
However, unless a separate electric distribution infrastructure is used, it
will present the same limitations as residential charging during blackouts,
with the added strain of long charging times. Additionally, as we find places
with high density have the lowest access risk, citing charging stations
presents a trade off between serving a large number of customers (placement in
high-density locations) and those with high access risk (generally low-density
locations). This same trade-off will be present to electricity utilities
allocating resources after power outages. The objectives of returning power
for the greatest number of people, and doing so in a way to not exacerbate
access risk will trade off against eachother.

In addition to infrastructure and transportation, access risk impacts other
dimensions of community resilience [45]. As a vulnerability of the
transportation sector, access risk prevents the normal operation of businesses
during blackouts, impacting economic resilience[46]. Additionally, social
resilience can be affected by access risk due to the ties between risk and
existing geographic inequality. Because inequality exists across and within
cities, associations between historically marginalized communities and
geographic inequality inside an urban area will exacerbate social disparities.

One way to adapt to access risk in the event of a long blackout is to directly
supply resources to vulnerable communities. During disruptive events, local
governments often put plans in place to help distribute essential goods, such
as food or water, that may be hard to obtain. Existing ways to distribute aid
can be altered to also support reducing access risk problems. This comes with
potential negative consequences which is that this policies increase the
pressure on support systems because current access issues are exacerbated by
vehicle electrification. Additionally, households facing access risk will be
more dependent on external support networks that will meet increased demand
which needs to be accounted for.

Our work aims to identify large scale interactions in a radically different
paradigm of individual mobility –assuming full electrification of personal
vehicles– which requires assumptions about human behavior and available
technology. For instance, the number of visits to each service will vary
across households, but a fixed number of trips still exposes the general
relation between geography and risk. Similarly, we considered 50% of the
energy remaining as the cutoff for access risk, and changing that threshold up
and down will increase or decrease the populations at risk. However, range
anxiety has been repeatedly identified in the context of electric vehicle
adoption [47] and is likely to have an effect on how this risk is perceived.
Additionally, daily commutes, evacuation attempts and leisure activities will
represent important loads for the energy reserves that have not been accounted
for in our study. Lastly, in order to estimate the V2G thresholds experienced
by each household, we assumed a fixed quantity of household electricity
consumption (Methods). We also note that access risk will still exist with
only Internal Combustion Engine Vehicles (ICEVs). Ongoing work is aimed at
understanding the specific change in access risk between ICEVs and BEVs.
However, an important takeway from our work is that access risk will still be
present under mass adoption of electric vehicles –regardless of the access
risk present with ICEV ownership.

## Methods

### Model Overview

We compute access risk at a census block group level based on weekly
electricity consumption required to access essential services which is a
function of geography. Weekly electricity consumption on mobility is estimated
from driving distance to the closest set of services and assuming average
vehicle efficiency (kWh/km) and uniform travel patters. We assumed complete
adoption of BEV for residential mobility and travel patterns for all
households consisting in two round trips to the closest supermarket, five to
the closest school and three to the closest park on a weekly basis. The number
of trips per week to each service was chosen to reflect basic mobility
patterns to access essential services. The number of trips has a linear
relationship with electricity consumption and a fixed number of trips allow us
to connect geographic inequality with access risk. Additionally, no commuting
was considered. Total distance driven for all services was computed for each
census block group and was multiplied by the nominal efficiency of the car to
obtain weekly battery consumption rates. No geographical or climatic variables
were considered to affect vehicle efficiency.

### Data

Combined Statistical Areas (CSA) were used to identify the limits of each
city. We chose a sample from the top 50 CSA by population according to the
numbers reported by the U.S. Census bureau as of July 1, 2022. We selected
most of the ten most populated areas and a sparse sample of the rest aiming
for variety in population size and geographic location. Each combined
statistical area is identified by a set of counties. For each county,
locations of all schools, supermarkets, parks and hospitals were obtained
using the Overpass API from OpenStreetMaps. Open Source Routing Machine [48]
was used to compute the closest driving distance to each type of amenity for
each Census Block Group in each CSA. The centroid of each Census Block Group
was used to compute this distance. Additionally, the 2021 5-Year survey data
from the American Community Survey was used to obtain population estimates and
car ownership rates per person per household for each census tract. Population
density was estimated using total land area information included in the TIGER
files. Aggregation of population density and car ownership rates per CSA was
done weighting each census tract by its population in order to account for
census tracts that represented a small percentage of the population.

### Inequality metric

The Equally Distributed Equivalent (EDE) metric was used to quantify and
compare inequality in the distributions of access for each geographical area.
Sheriff and Maguire [49] introduced the benefits of Equally distributed
equivalents for quantifying risk and burden inequality in addition to
resources. In particular, the EDE metrics are beneficial because they capture
level and dispersion of the distribution, in contrast to popular metrics like
the Gini index which is only concerned with dispersion. The Kolm–Pollak EDE
for a distribution $X$ is defined according to Equation 1 and has been used
for measuring inequalities in urban systems [35].

$\Xi(X)=-\frac{1}{\kappa}ln\left[\frac{1}{N}\sum_{i=1}^{N}e^{\kappa
x_{i}}\right]$ (1)

Where $\Xi$ is the Equally distributed equivalent, $N$ is the sample size and
$x_{i}$ is the $i^{t}h$ value in the distribution of $X$. The inequality
aversion parameter $\kappa$ is input by the user and represents a weight for
dispersion in the distribution. Atkinson [50] propose to calculate $\kappa$
according to Equation 2, where $\epsilon$ is a user input parameter that whose
absolute value is typically between 1 and 2 and its sign determines the nature
of the distribution. Positive values are used to quantify dispersion in
desirable quantities and negative values are used for burdens. The value of
$\epsilon$ used in this work is 1.75.

$\kappa=\frac{\sum_{i=1}^{N}x_{i}}{\sum_{i=1}^{N}x_{i}^{2}}\epsilon$ (2)

### Technology

Base scenario electric vehicle battery capacity and efficiency were chosen to
correspond to a Tesla Model Y (2022) according to the EV-Database website
[32]. In particular, this means 57.5 kWh of usable battery (approximated to
60kWh to facilitate the generation of scenarios with increased/reduced
battery) and a range of 350km resulting in an average efficiency of 5.83
km/kWh. The database was also used for estimating a cost per kWh of additional
battery in the current market. We found that below $80,000 in price, vehicles
showed a trend of $1,000 per additional kWh. Above that price, additional
increases were not translated to increase capacity.

Household weekly electricity consumption was estimated using yearly
residential consumption averages by end-use from the Energy Information
Administration of the U.S[51]. The end-uses considered for daily V2G load were
water heating, air conditioning, lighting and clothes dryer and the average
consumption values were 28.14 kWh, 34.84 kWh, 21.24 kWh and 9.34 kWh per week
respectively.

## References

  * [1] Jenn, A.  Emissions benefits of electric vehicles in Uber and Lyft ride-hailing services.  _Nature Energy_ 5, 520–525, DOI: 10.1038/s41560-020-0632-7 (2020).
  * [2] Camilleri, S. F. _et al._ Air quality, health and equity implications of electrifying heavy-duty vehicles.  _Nature Sustainability_ 1–11, DOI: 10.1038/s41893-023-01219-0 (2023).
  * [3] Liang, X. _et al._ Air quality and health benefits from fleet electrification in china.  _Nature Sustainability_ 2, 962–971, DOI: 10.1038/s41893-019-0398-8 (2019).
  * [4] Chang, S. Y. _et al._ Electric vehicle fleet penetration helps address inequalities in air quality and improves environmental justice.  _Communications Earth & Environment_ 4, 1–15, DOI: 10.1038/s43247-023-00799-1 (2023).
  * [5] Pan, S. _et al._ Impacts of the large-scale use of passenger electric vehicles on public health in 30 US. metropolitan areas.  _Renewable and Sustainable Energy Reviews_ 173, 113100, DOI: 10.1016/j.rser.2022.113100 (2023).
  * [6] Popovich, N. D., Rajagopal, D., Tasar, E. & Phadke, A.  Economic, environmental and grid-resilience benefits of converting diesel trains to battery-electric.  _Nature Energy_ 6, 1017–1025, DOI: 10.1038/s41560-021-00915-5 (2021).
  * [7] Ravi, S. S. & Aziz, M.  Utilization of electric vehicles for vehicle-to-grid services: Progress and perspectives.  _Energies_ 15, 589, DOI: 10.3390/en15020589 (2022).
  * [8] Tan, K. M., Ramachandaramurthy, V. K. & Yong, J. Y.  Integration of electric vehicles in smart grid: A review on vehicle to grid technologies and optimization techniques.  _Renewable and Sustainable Energy Reviews_ 53, 720–732, DOI: 10.1016/j.rser.2015.09.012 (2016).
  * [9] Ren, Y. _et al._ Hidden delays of climate mitigation benefits in the race for electric vehicle deployment.  _Nature Communications_ 14, DOI: 10.1038/s41467-023-38182-5 (2023).
  * [10] Milovanoff, A., Posen, I. D. & MacLean, H. L.  Electrification of light-duty vehicle fleet alone will not meet mitigation targets.  _Nature Climate Change_ 10, 1102–1107, DOI: 10.1038/s41558-020-00921-7 (2020).
  * [11] Zhang, C., Zhao, X., Sacchi, R. & You, F.  Trade-off between critical metal requirement and transportation decarbonization in automotive electrification.  _Nature Communications_ 14, DOI: 10.1038/s41467-023-37373-4 (2023).
  * [12] Trost, J. N. & Dunn, J. B.  Assessing the feasibility of the inflation reduction act’s ev critical mineral targets.  _Nature Sustainability_ 6, 639–643, DOI: 10.1038/s41893-023-01079-8 (2023).
  * [13] Richter, J. L.  A circular economy approach is needed for electric vehicles.  _Nature Electronics_ 5, 5–7, DOI: 10.1038/s41928-021-00711-9 (2022).
  * [14] Zhang, H. _et al._ Life cycle environmental impact assessment for battery-powered electric vehicles at the global and regional levels.  _Scientific Reports_ 13, DOI: 10.1038/s41598-023-35150-3 (2023).
  * [15] Powell, S., Cezar, G. V., Min, L., Azevedo, I. M. L. & Rajagopal, R.  Charging infrastructure access and operation to reduce the grid impacts of deep electric vehicle adoption.  _Nature Energy_ 7, 932–945, DOI: 10.1038/s41560-022-01105-7 (2022).
  * [16] Hussain, A. & Musilek, P.  Resilience enhancement strategies for and through electric vehicles.  _Sustainable Cities and Society_ 80, 103788, DOI: 10.1016/j.scs.2022.103788 (2022).
  * [17] Raman, G., Raman, G. & Peng, J. C.-H.  Resilience of urban public electric vehicle charging infrastructure to flooding.  _Nature Communications_ 13, 3213, DOI: 10.1038/s41467-022-30848-w (2022).
  * [18] Nunes, A., Woodley, L. & Rossetti, P.  Re-thinking procurement incentives for electric vehicles to achieve net-zero emissions, DOI: 10.1038/s41893-022-00862-3 (2022).
  * [19] Adderly, S. A., Manukian, D., Sullivan, T. D. & Son, M.  Electric vehicles and natural disaster policy implications.  _Energy Policy_ 112, 437–448, DOI: 10.1016/j.enpol.2017.09.030 (2018).
  * [20] Almutairi, A.  Impact Assessment of Diverse EV Charging Infrastructures on Overall Service Reliability.  _Sustainability_ 14, 13295, DOI: 10.3390/su142013295 (2022).
  * [21] Cano, Z. P. _et al._ Batteries and fuel cells for emerging electric vehicle markets.  _Nature Energy_ 3, 279–289, DOI: 10.1038/s41560-018-0108-1 (2018).
  * [22] Weiss, M., Cloos, K. C. & Helmers, E.  Energy efficiency trade-offs in small to large electric vehicles.  _Environmental Sciences Europe_ 32, DOI: 10.1186/s12302-020-00307-8 (2020).
  * [23] Logan, T. M. & Guikema, S. D.  Reframing resilience: Equitable access to essential services.  _Risk Analysis_ 40, 1538–1553, DOI: 10.1111/risa.13492 (2020).
  * [24] International Energy Agency.  Global CO2 emissions by sector, 2019-2022.  https://www.iea.org/data-and-statistics/charts/global-co2-emissions-by-sector-2019-2022.
  * [25] U.S. Environmental Protection Agency.  Inventory of U.S. Greenhouse Gas Emissions and Sinks: 1990-2021 (2023).
  * [26] U.S. Energy Information Administration.  U.S. Energy-Related Carbon Dioxide Emissions (2022).
  * [27] Winkler, L., Pearce, D., Nelson, J. & Babacan, O.  The effect of sustainable mobility transition policies on cumulative urban transport emissions and energy demand.  _Nature Communications_ 14, DOI: 10.1038/s41467-023-37728-x (2023).
  * [28] Wolinetz, M., Axsen, J., Peters, J. & Crawford, C.  Simulating the value of electric-vehicle–grid integration using a behaviourally realistic model.  _Nature Energy_ 3, 132–139, DOI: 10.1038/s41560-017-0077-9 (2018).
  * [29] International Energy Agency.  Net Zero by 2050 (2021).
  * [30] International Energy Agency.  World Energy outlook (2022).
  * [31] World Business Council for Sustainable Development.  Value framework for sustainable charging infrastructure (2021).
  * [32] Electric Vehicle Database.  https://ev-database.org/.  Accessed: 2023-03-30.
  * [33] Macmillan, M. _et al._ Shedding light on the economic costs of long-duration power outages: A review of resilience assessment methods and strategies.  _Energy Research & Social Science_ 99, 103055, DOI: 10.1016/j.erss.2023.103055 (2023).
  * [34] U.S. Energy Information Administration (EIA).  Hurricane ida caused at least 1.2 million electricity customers to lose power.  https://www.eia.gov/todayinenergy/detail.php?id=49556.  Accessed: 2024-02-06.
  * [35] Logan, T. M., Anderson, M. J., Williams, T. G. & Conrow, L.  Measuring inequalities in urban systems: An approach for evaluating the distribution of amenities and burdens.  _Computers, Environment and Urban Systems_ 86, 101590, DOI: 10.1016/j.compenvurbsys.2020.101590 (2021).
  * [36] Scheba, A., Turok, I. & Visagie, J.  Inequality and urban density: Socio-economic drivers of uneven densification in cape town.  _Environment and Urbanization ASIA_ 12, S107–S126, DOI: 10.1177/0975425321998026 (2021).
  * [37] Moody, J., Farr, E., Papagelis, M. & Keith, D. R.  The value of car ownership and use in the united states.  _Nature Sustainability_ 4, 769–774, DOI: 10.1038/s41893-021-00731-5 (2021).
  * [38] Grengs, J.  Job accessibility and the modal mismatch in detroit.  _Journal of Transport Geography_ 18, 42–54, DOI: 10.1016/j.jtrangeo.2009.01.012 (2010).
  * [39] Frederick, C. & Gilderbloom, J.  Commute mode diversity and income inequality: an inter-urban analysis of 148 midsize us cities.  _Local Environment_ 23, 54–76, DOI: 10.1080/13549839.2017.1385001 (2018).
  * [40] Giuliano, G. & Dargay, J.  Car ownership, travel and land use: a comparison of the us and great britain.  _Transportation Research Part A: Policy and Practice_ 40, 106–124, DOI: 10.1016/j.tra.2005.03.002 (2006).
  * [41] Kim, S. & Haskell, H.  Willingness to Use Vehicle-to-Everything (V2X) - 2021 Pilot Survey.  https://coloradosmart.city/wp-content/uploads/2022/01/V2X_PilotSurveyCU_2021.pdf (2021).  Accessed: 2024-02-13.
  * [42] U.S. Internal Revenuew Service.  Inflation reduction act 2022.  https://www.irs.gov/inflation-reduction-act-of-2022.  Accessed: 2023-12-20.
  * [43] Nykvist, B. & Nilsson, M.  Rapidly falling costs of battery packs for electric vehicles.  _Nature Climate Change_ 5, 329–332, DOI: 10.1038/nclimate2564 (2015).
  * [44] Cano, Z. P. _et al._ Batteries and fuel cells for emerging electric vehicle markets.  _Nature Energy_ 3, 279–289, DOI: 10.1038/s41560-018-0108-1 (2018).
  * [45] Cutter, S. L. _et al._ A place-based model for understanding community resilience to natural disasters.  _Global Environmental Change_ 18, 598–606, DOI: 10.1016/j.gloenvcha.2008.07.013 (2008).
  * [46] Rose, A.  Defining and measuring economic resilience to disasters.  _Disaster Prevention and Management: An International Journal_ 13, 307–314 (2004).
  * [47] Chakraborty, P. _et al._ Addressing the range anxiety of battery electric vehicles with charging en route.  _Scientific Reports_ 12, DOI: 10.1038/s41598-022-08942-2 (2022).
  * [48] Project OSRM: Open Source Routing Machine.  https://project-osrm.org/.  Accessed: 2023-12-22.
  * [49] Sheriff, G. & Maguire, K. B.  Health risk, inequality indexes, and environmental justice.  _Risk Analysis_ 40, 2661–2674, DOI: 10.1111/risa.13562 (2020).
  * [50] Atkinson, A. B.  On the measurement of inequality.  _Journal of Economic Theory_ 2, 244–263, DOI: 10.1016/0022-0531(70)90039-6 (1970).
  * [51] U.S. Energy Information Administration (EIA).  Annual household site end-use consumption in the United States—totals and averages.  https://www.eia.gov/consumption/residential/data/2020/c&e/pdf/ce3.1.pdf (2020).  Accessed: 2024-02-13.

## Acknowledgements (not compulsory)

Y.E. would like to thank and acknowledge funding from the Edward P. Fitts
Doctoral Scholarship.

## Author contributions statement

Y.E. and B.R. conceived the idea; Y.E. conducted the analysis and experiments;
Y.E. and B.R. drafted, reviewed and edited the manuscript. B.R. advised and
funded.


# Crowdsourcing Controller - Utilizing Reliable Agents in a Multiplayer Game
††thanks: This paper is partially supported by the EU Horizon 2020 research
and innovation programme under the MSCA grant No. 893667

1st Kacper Kenji Lesniak  University of Copenhagen and Dynasty Studios
Copenhagen, Denmark
<EMAIL_ADDRESS>2nd Maria Maistro  University of Copenhagen
Copenhagen, Denmark
<EMAIL_ADDRESS>
###### Abstract

This paper presents a new use case for continuous crowdsourcing, where
multiple players collectively control a single character in a video game.
Similar approaches have already been proposed, but they suffer from certain
limitations: (1) they simply consider static time frames to group real-time
inputs from multiple players; (2) then they aggregate inputs with simple
majority vote, i.e., each player is uniformly weighted.

We present a continuous crowdsourcing multiplayer game equipped with our
Crowdsourcing Controller. The Crowdsourcing Controller addresses the above-
mentioned limitations: (1) our Dynamic Input Frame approach groups incoming
players’ input in real-time by dynamically adjusting the frame length; (2) our
Continuous Reliability System estimates players’ skills by assigning them a
reliability score, which is later used in a weighted majority vote to
aggregate the final output command.

We evaluated our Crowdsourcing Controller offline with simulated players and
online with real players. Offline and online experiments show that both
components of our Crowdsourcing Controller lead to higher game scores, i.e.,
longer playing time. Moreover, the Crowdsourcing Controller is able to
correctly estimate and update players’ reliability scores.

###### Index Terms:

Crowdsourcing, Multiplayer, Reliability system

##  I Introduction

Crowdsourcing systems are fascinating and powerful tools that can be utilized
in a variety of use cases [3]. Thanks to wide access to a stable and fast
network connection, it is possible now to develop those systems for real-time
applications, i.e., continuous crowdsourcing. In recent years, research has
been dedicated to real-time and online crowdsourcing systems, with use cases
such as video captioning, robot controls, audio processing, and assistive
technology [8].

Another topic that emerged in recent years are video game streams, where
spectators become active players by voting on certain gameplay choices or even
taking direct control of the game. Recent research has found that watching a
stream of a video game is more about social interaction than the game content
itself [2]. This type of games, where larger crowds play and cooperate
together through aggregated network input are described as Emergent
Multiplayer Games (EMG) [14]. The popularity of these events shows that there
is a market for new, innovative ways in which players can cooperate in game
contexts. Although research has been put into how a multiplayer voting
mechanisms can work from a gameplay perspective [14], there is little insight
into merging crowdsourcing approaches with multiplayer video game controllers.

In this paper, we explore continuous crowdsourcing approaches in the context
of multiplayer games, in which players make collective decisions regarding the
gameplay. We present a new use case for continuous crowdsourcing, where
multiple players collectively control a single character in a video game.
Similar approaches have already been proposed in different contexts, but they
suffer from certain limitations. They consider static frames to group inputs
from multiple users and they do not utilize approaches commonly implemented in
crowdsourcing systems to estimate players reliabilities when aggregating
inputs.

The contribution of this paper is twofold. The first part is a simple video
game with an innovative mechanic that lets multiple network players control
one character through an aggregated input. The second part is our
Crowdsourcing Controller, which controls how to: (1) group real-time inputs
coming from multiple users into frames; (2) aggregate inputs within a frame
and issue a single output command. We propose two novel approaches to
aggregate and process crowdsourced inputs: the Dynamic Input Frame to group
inputs and the Continuous Reliability System to aggregate them. The former
aims at detecting a precise moment in time when the input should be issued to
the game, while the latter focuses on recognizing and utilizing reliable
players during the play session. A combination of offline, AI-based
experiments, as well as online experiments, with $9$ real players, has been
conducted to gather the data needed for the analysis and evaluation of the
proposed solutions.

The paper is organized as follows: related work on crowdsourcing and online
multiplayer games (§ II); the online game (§ IV); our Crowdsourcing Controller
(§ IV); experimental evaluation (§ V); conclusions and future work (§ VI).

##  II Related Work

First, we describe crowdsourcing, its definitions and most prevalent use cases
(§ II-A). Then we present related work on Emergent Multiplayer Games (EMG), a
type of online video streams in which viewers have direct control over the
game through voting mechanisms (§ II-B).

###  II-A Crowdsourcing

The term crowdsourcing firstly appeared in $2006$ with the meaning of “taking
a function once performed by employees and outsourcing it to an undefined (and
generally large) network of people in the form of an open call” [6]. Over the
years, this idea has gained much interest both in industry and academia and
became an umbrella term that encompasses many different use cases, as
distributed-human-intelligence tasks, knowledge discovery and management, etc.
[3]. One of the most famous examples is Amazon Mechanical Turk, a
crowdsourcing platforms where requesters can submit small jobs, such as image
labeling, called Human Intelligence Tasks (HITs), which are performed by a
remote community of workers. In recent years, researchers explored novel ideas
to use crowdsourcing such as software development [11] and even writing
fiction [7].

A persistent problem in crowdsourcing is how to engage crowd workers when
performing HITs. Usually crowd workers are paid a small amount of money [10],
which does not guarantee that workers perform to the best of their ability,
even if a fair payment can be helpful [5, 15]. Another way of engaging workers
is through gamification: in [1] a novel platform is designed, where users
playing an online game were labeling images while competing against each
other.

Providing engaged and motivated workers does not always guarantee that their
answers will be correct. Therefore each HIT is usually assigned to multiple
crowd workers, whose inputs are aggregated with statistical methods. The
simplest way to aggregate multiple answers is by Majority Vote (MV), i.e., the
answer that is the most popular among the workers is predicted to be correct.
This, however, does not take into consideration differences in workers’
expertise.

Expectation Maximization (EM) algorithm has been used as early as in [4] to
tackle this problem. EM iteratively estimates the probability that a worker
correctly completes a set of HITs (this probability is called reliability in §
IV). Some limitations of EM are that it needs to run offline on the whole
dataset and it is costly due to its iterative nature. Both of these arguments
prevent EM application in real-time scenarios.

A relatively new type of crowdsourcing is Continuous crowdsourcing [8] which
engages workers over long periods and allows them to maintain the context of
the task. Moreover the tasks change over time reacting to workers’ input. The
responses from the workers are processed in so-called input mediators that
aggregate them to produce the system output. Continuous crowdsourcing has been
applied to a variety of use cases, for example, real-time continuous
video/audio captioning to collect activity recognition labels from video
stream [9], and a robot control system able to react to unforeseen
circumstances[8].

Continuous crowdsourcing requires to dynamically estimate workers’
reliabilities. Such calculations need to be done in real-time, as workers’
input is immediately applied to the task at hand and different reliabilities
affect the votes aggregation. In [13] a modified version of the EM algorithm
is used in real-time to estimate workers’ reliabilities. The authors, however,
do not mention the time complexity of this approach, and using an iterative
method in real-time scenarios might not be optimal. In this paper, we propose
a method of calculating workers reliabilities in a real-time scenario without
the need to perform high complexity calculations.

###  II-B Video Games Controlled by Multiple Players

There is a growing trend of multiplayer video games, where multiple network
users control a single instance of a video game streamed through an online
platform. One of the first, and to this day the most popular event using this
approach was Twitch Plays Pokemon, where thousands of players watched a video
stream of an ongoing Pokemon Red game while issuing commands through the chat.
The game creators defined two modes: anarchy, all inputs from all players are
forwarded to the game; and democracy, all inputs are considered as votes,
counted and then forwarded to the game, similarly to input mediators (§ II-A).
The democracy mode resulted in a more conservative and slower behavior of the
character, but allowed the players to beat the hardest challenges.

Twitch Plays Pokemon is one of the earliest examples of an Emergent
Multiplayer Game (EMG) [14], a video game in which large crowds play
collectively through a network voting system. EMGs utilize a set of relatively
simple game mechanics and subsystems, that when controlled by large crowds
result in creative and unique ways of playing. They encourage coordination and
self-organization of the crowd, both in-game and outside of it. Explored
mechanics usually use a voting system that gathers input from a large crowd
and applies the results after a certain delay. The time reserved for
collecting the input, however, makes the system not adequate for fast-paced,
real-time scenarios. We address this challenge using our Crowdsourcing
Controller.

##  III System Overview and Use Case

Our continuous crowdsourcing system is presented in Figure 1. The central part
of the system is a simple video game (described in § III-A). Players issue
inputs through the network as if they were the only player in control of the
game character. The input is aggregated and processed by the Crowdsourcing
Controller. The output of the system is a singular game command, whose type
and issue time are calculated based on all the inputs sent over the network.
The aggregated game command is applied with immediate effect, which influences
the state of the game and is seen by all players on their screens. The source
code of the implementation of our system is publicly
available111https://github.com/KacperKenjiLesniak/crowded-dinosaur.

Figure 1: General system overview with technologies used for implementation

###  III-A Use case: a Simple Video Game

Our Crowsourcing Controller controls a character in a simple video game which
we describe next. We choose this use case because: (1) crowd workers are
generally familiar with simple controls as well as typical challenges
presented in video games; (2) we wanted to explore the presented crowdsourcing
system as part of a novel gameplay mechanic; (3) gamification has proven to be
a viable method to engage users in a variety of tasks [12] which can help with
keeping workers motivated throughout the course of experiments.

Figure 2: Crowd controlled dinosaur (black) and player dinosaurs (different
colors) jumping over an obstacle

The implemented game is a clone of a well-known mini-game played in Chrome
(see Figure 2). Players are assigned their own dinosaur character of a
different color. The crowd-controlled dinosaur which corresponds to the
aggregated output is represented by a black dinosaur. Players’ dinosaurs
behave like “ghosts” and they serve purely as a visual clue to the players
(Figure 2). Thanks to them the players can see that their input is being
processed and is working as intended, even when the crowdsourced dinosaur is
not reflecting their actions at the current moment. The players also see each
other characters, which can enhance engagement and competition, and it
provides further visual feedback. Players seeing the black dinosaur impacting
an obstacle, even when they issued a correct input, can feel frustrated and
believe that the system is not working. Seeing that others failed to issue the
correct input can help alleviate this issue. Moreover, seeing all other
characters can lead to players quickly realizing that they are issuing the
inputs out of sync.

We chose this game because it meets a specific set of requirements: the input
comes in real-time and jumping or crouching at the right moment is crucial for
progression; the input is discrete (short jump, long jump, crouch); there is a
player score which can be used as an estimate of player skills, and the game
is widely known and simple to play.

##  IV Crowdsourcing Controller

We describe the Crowdsourcing Controller next, which is responsible for
analyzing the inputs coming from the players, aggregating them, and issuing a
single command that best represents the combined inputs.

Let us consider a system where four players are able to issue two types of
input - $1$ and $2$. The example in Figure 3 presents a certain moment in
time, during which the system receives four inputs, each coming from a
different player, three of them are of type $1$ and one is of type $2$.

Figure 3: Inputs of $2$ types issued by $4$ players in a short period of time

This situation poses two challenges. The first challenge relates to the
definition of the input frame, i.e., the time window where all inputs
correspond to the players’ reaction to the same game event, for example an
obstacle appearing on screen. In the example in Figure 3, all $4$ inputs are
issued at a similar time, so it is reasonable to consider them in the same
group or input frame. As for when the input frame should be analyzed further
by the system, in the presented example it is reasonable to do it immediately
after receiving the last input. Moreover, since no more inputs are issued
after the last one, all inputs can be grouped and processed by the system for
further analysis. However, it is not possible to make similar decisions in a
real-time setting without knowing what comes next. Section IV-A describes how
we tackle this problem with dynamic and static input frames.

The second challenge relates to the aggregation approach, i.e., how to combine
multiple inputs in a single one. In the presented example, three players voted
for input $1$ and one player for input $2$, so according to majority vote, the
output of the system should be $1$. However, majority vote treats all payers
equally and does not consider the expertise of each player, e.g., the vote of
an expert or reliable player can weight more than the vote from a beginner and
lead to a more accurate aggregated score. Section IV-B describes how we
estimate such weights, called reliabilities.

Figure 4: Flow of players’ inputs through different input mediators

The above mentioned challenges are handled by input mediators, illustrated in
Figure 4. Input mediators are components of the crowdsourcing system that: (1)
process the input stream coming in real-time from several players and group
inputs into frames; and (2) produce the final aggregated output that is issued
to the game. These two components together define our Crowdsourcing
Controller.

###  IV-A Dynamic and static Input Frames

The first component of the Crowdsourcing Controller groups continuous input
into input frames, which contain no more than a single input for each player
(a missing input from a player is represented as $0$). A frame represents a
short moment in time in which the users issue the commands which can result in
a specific behavior of the system. We use $2$ types of input frames, static
and dynamic, which we describe next.

The Static Input Frames [8] slice the input stream into frames of a constant
length. In case of multiple inputs from the same player in a single frame, the
system uses the most recent one. One limitation of this approach is how to
define the frame length: too short frames will wrongly group inputs related to
the same event in different frames, too long frames will negatively affect the
system responsiveness.

To address this limitation, we propose the Dynamic Input Frame approach, where
the frame length is dynamically defined based on players’ inputs. The system
forms a queue of incoming inputs and decides in real-time when to slice the
input frame. The inputs in the queue have a specific Time To Live (TTL), after
which they are discarded. When the number of inputs in the queue surpasses a
threshold (T), the system begins the procedure of defining the frame. It waits
for a time equal to half of the TTL, then gathers all the inputs in the queue,
passes them to the aggregator component, and clears the input queue. It is
crucial to correctly estimate TTL, as it creates a trade-off between the
responsiveness of the system and its accuracy, as illustrated in Figure 5.

When the Dynamic Input Frame is used together with the Continuous Reliability
System (§ IV-B), instead of using the number of inputs to compare to the
threshold (T) we calculate a weighted sum using players’ reliabilities, so
that reliable players have more control over when the input frame is issued.

Figure 5: Dynamic frames, threshold (T) of 2 players out of 4 marks the center
of the frame: a) too short duration, part of input ignored; b) too long
duration, unresponsive system; c) correct duration for the situation

###  IV-B Inputs Aggregator

The second component of the Crowdsourcing Controller gets all the inputs
within the input frame and returns a single aggregated input. We consider two
approaches to aggregate inputs: (1) Majority Vote (MV); and (2) our Continuous
Reliability System.

MV simply selects the most popular input among players. Ties are arbitrarily
broken with a predefined order of preferred inputs. Note that no or missing
input also counts as a valid input, because sometimes the correct action is
not issuing any commands at all.

The Continuous Reliability System uses weighted MV, where the votes are
weighted according to the reliability score of each player, i.e., an estimate
of the ability of each player to issue correct inputs. Next, we describe how
to compute and update reliabilities in real-time.

Inspired by the idea of Collective Intelligence, the Continuous Reliability
System rewards players that issue the same input as the majority and punishes
those that vote for unpopular options. The system starts by aggregating inputs
with MV, i.e., uniform reliabilities. Every time the system receives an input
frame, the reliability scores are updated proportionally to how popular the
players’ votes are. For example, when $95\%$ of the players vote for input A
and $5\%$ vote for input B, the reliabilities of the players that voted for
option B will drop significantly, compared to a situation where a split
between A and B is $60\%$ to $40\%$. After a certain number of iterations,
reliability scores might change the final output: the aggregated input is not
the most popular one, but a less popular option can be chosen because of the
high reliability of the players who issued that command. This system is
completely decoupled from the game itself, it does not need to be aware of
whether an input was good or bad based on the game state, which makes it
easier to apply it to other scenarios in the future.

Next we formally describe the Continuous Reliability algorithm (Algorithm 1).
Let $\mathcal{I}=\\{0,1,\dots K-1\\}$ be the list of all possible inputs
issued by players and $N_{p}$ be the number of players. A single input frame
$\mathcal{F}$ is an $N_{p}$-dimensional vector with values from $\mathcal{I}$,
i.e., the votes of the players within the time frame. $R_{j}$ is the
reliability score of player $j$ and $R$ is the $N_{p}$-dimensional vector
containing all current reliability scores.

The Continuous Reliability algorithm is recurrent. At the start, each player
is assigned the same reliability score of $1$, then at each iteration we
compute $\Delta R$, i.e., the vector of reliability changes that is added to
update reliability scores.

Consider a fixed iteration with input frame $\mathcal{F}$. We define
$\mathcal{V}=\\{v_{0},v_{1},\dots v_{K-1}\\}$ as a $K$-dimensional vector
counting the number of votes for each input in $\mathcal{F}$. We say that an
input $j$ is viable when it receives “enough” votes, i.e.,
$v_{j}\geq\gamma*v_{max}$, where $\gamma$ is an agreement threshold and
$v_{max}=\text{max}(\mathcal{V})$, i.e., the maximum number of votes achieved
by a single input. This handles cases when the votes are evenly split between
more than one inputs: we do not want to decide that only one input is correct,
but we will consider all viable votes when updating player reliabilities.

If all inputs in $\mathcal{F}$ are viable we do not change reliabilities,
because all players selected a potentially correct input. If there are non-
viable inputs, we proceed to compute $\Delta R$.

First we treat non-viable inputs in $\mathcal{F}$, intuitively they should
affect $\Delta R$ in a proportional way: the less popular an input, the lower
the player reliability who issued that. We define this as follows:

$\Delta R_{{j}_{nv}}=-\frac{v_{max}}{v_{\mathcal{F}_{j}}}*\delta$ (1)

where $j$ is a player who issued a non-viable input; $v_{\mathcal{F}_{j}}$ is
the number of votes received by the input issued by player $j$; and
$\delta\geq 0$ is the reliability step coefficient, which controls how much a
viable or non-viable input affects the reliability score. The higher $\delta$,
the faster the system will adjust the reliabilities. The current reliability
of player $j$, $R_{j}$, is updated by adding (subtracting) the quantity in
Equation (1).

Note that the reliability scores might fall below $0$, which means that
unreliable players will in fact decrease the number of votes for the input
they choose. This mechanism can be effective when dealing with players that
issue wrong inputs on purpose, as they still unknowingly help the system in
selecting the correct input. On the other side, a limitation of this mechanism
arises when the same player keeps on issuing wrong inputs, e.g., inactivity,
network lag, etc. If the player resumes the game and starts issuing correct
inputs, his/her very low reliability will disrupt the voting, since weighted
MV will heavily favor options not chosen by this player. Therefore, we
introduce a lower bound $\omega\geq 0$, which clamps the change of the
reliability if it brings the total player reliability below $-\omega$:

$\Delta
R_{{j}_{nv}}=\begin{cases}\frac{v_{max}}{v_{\mathcal{F}_{j}}}*\delta,&\text{if
}R_{j}+\Delta R_{{j}_{nv}}\geq-\omega\\\ \omega-
R_{j},&\text{otherwise}\end{cases}$

where $R_{j}$ is the current reliability score for player $j$. The opposite
problem is not a concern for the upper bound. For players to have a high
reliability score, they not only have to be consistent in issuing the input,
but also other players have to vote similarly to them. This makes a situation
where one of the players has a significantly larger reliability score than
others very unlikely.

Then, we treat viable inputs in $\mathcal{F}$. Again they should affect
$\Delta R$ in a proportional way: the more popular an input, the higher the
player reliability who issued that. We do this as follows:

$\Delta R_{{j}_{v}}=v_{\mathcal{F}_{j}}*\frac{\sum_{k\in nv}\Delta
R_{k}}{\sum_{k\in v}v_{\mathcal{F}_{k}}}$ (2)

where the first factor $v_{\mathcal{F}_{j}}$ is the number of votes received
by the input issued by player $j$ and the second factor is a normalization
factor. The normalization factor is computed as the ratio of the sum $\Delta
R$ for non-viable inputs and the sum of $v_{\mathcal{F}_{k}}$ for the viable
inputs. The normalization factor is needed to guarantee that the sum of
reliabilities remains constant and equal to the number of players, which is
needed by weighted MV.

Input:  Players input frame $\mathcal{F}$, current reliabilities
$R=[R_{0},\dots R_{N_{p}-1}]$

Result: Vector of reliabilities updates $\Delta R=[\Delta R_{0},\dots\Delta
R_{N_{p}-1}]$

$\Delta R=[0,0,\dots 0]_{N_{p}}$

$v_{\mathcal{F}_{j}}=\sum_{k}(\mathcal{F}_{j}==\mathcal{F}_{k})$ // Number of
votes for the input issued by player j

if  _$\forall j\in viable(v_{\mathcal{F}_{j}})$_ then

return _$\Delta R$_

end if

// First pass - non-viable votes

for  _not $viable(v_{\mathcal{F}_{j}})\in\mathcal{V}$_ do

$\Delta R_{j}=-\frac{v_{max}}{v_{\mathcal{F}_{j}}}*\delta$

if  _$R_{j}+\Delta R_{j}\leq-\omega$_ then

$\Delta R_{j}=\omega-R_{j}$

end if

end for

$norm=\frac{\sum_{k\in nv}\Delta R_{k}}{\sum_{k\in v}v_{\mathcal{F}_{k}}}$ //
Viable votes normalization factor

// Second pass - viable votes

for  _$viable(v_{\mathcal{F}_{j}})\in\mathcal{V}$_ do

$\Delta R_{j}=v_{\mathcal{F}_{j}}*norm$

end for

return _$\Delta R$_

Algorithm 1 Calculating the reliability updates

##  V Experimental Evaluation

Next we evaluate the performance of the Crowdsourcing Controller, i.e., input
frames and input aggregators.

###  V-A Experimental Setup

The Crowdsourcing Controller depends on a number of parameters. For static
input frames we need to set only the frame length. For dynamic input frames we
need to set the TTL and the frame threshold T, i.e., the number of inputs from
different users needed to determine the centre of the dynamic time window (see
Figure 5). We set TTL to $300$ms, so on average the time between issuing an
input and seeing a response on the screen is around $150$ms. Threshold $T=0.5$
was set so that half of the inputs marked the centre of the dynamic frame. For
the continuous reliability system, we need to estimate the agreement threshold
$\gamma\in[0,1]$, i.e., the proportion of votes needed to define a viable
input, and the reliability step coefficient $\delta\in[0,1]$, which determines
the speed in updating reliabilities. The parameters have been fine-tuned for
offline and online experiments during development and used consistently to
compare the system under the same circumstances. The values were chosen so
that the learning process of the system can stabilize in few minutes, as
longer sessions could cause players’ fatigue. The values used were
$\gamma=0.8$ (offline), $\gamma=0.6$ (online) and $\delta=0.05$. The lower
bound $\omega$ is set to $0.5$.

For the input frame component, we consider the simple static input frame
approach as our baseline opposed to our dynamic input frame approach. For the
inputs aggregator component we consider MV as baseline opposed to our
Continuous Reliability System. In the experiments we consider $4$ different
system configurations: (1) static frame with Continuous Reliability System;
(2) Dynamic Input Frame with Continuous Reliability System; (3) Static frame
with MV; (4) Dynamic Input Frame with MV.

For the evaluation, there is no available ground-truth, i.e., single correct
sequence of inputs, that we can use to evaluate the Crowdsourcing Controller.
This is because the game is dynamic and there is more than one way of beating
obstacles, e.g., some obstacles can be beaten with both short and long jumps.
Therefore we use the game score as evaluation measure. The game score
represents how long it takes for the players to lose the game, by issuing the
wrong type of input or providing it at the incorrect time. The higher the game
score the better.

Additionally we consider EM [4] as our reference or “ground-truth”. Due to its
iterative definition, EM can not be run online, therefore it is run offline
when all data are gathered. Then we compute the agreement between one of the
$4$ configurations and EM, i.e., the accuracy with the inputs issued by a
configuration and the inputs predicted by EM as ground-truth. The higher the
agreement the better.

###  V-B Offline Experiment with Synthetic Players

In this experimental scenario we evaluate the Crowdsourcing Controller in an
artificial setting with synthetic players. Players are replaced with AIs
designed to represent humans of different skills. The Perfect AI always issues
a correct input at each time-stamp during the game. All other AIs are defined
by adding noise to the perfect AI behaviour. This is done with $3$ parameters
representing: noise, noise shift and chance of error. We generated a total of
$14$ different AIs: $1$ Perfect, $3$ Good, $4$ Bad, $1$ Shifted (-), $2$
Shifted (+), $1$ Majorly Shifted (-), $1$ Confused, and $1$ Random. Each AI
was individually tested 3 times and their mean scores are in Table I.

TABLE I: AIs individual game scores Name | Average Score | Max Score
---|---|---
Perfect | 1182 | 1353
Good | 250 | 488
Bad | 63 | 66
Confused | 23 | 28
Random | 48 | 64
Shifted (-) | 164 | 398
Shifted (+) | 27 | 30
Majorly Shifted (-) | 98 | 175
Figure 6: 14 AIs scores with different system configurations TABLE II: 14 AIs experiment results Name |  | Aggrement
---
w/ EM predictions
| Mean
---
score
| Max
---
score
Dynamic Reliability | 0.9937 | 852 | 1211
Dynamic Majority | 0.9915 | 474 | 981
Static Reliability | 0.9553 | 309 | 610
Static Majority | 0.9714 | 94 | 277

The comparison of scores achieved by different system configurations across
$6$ consecutive trials is presented in Figure 6 and in Table II. We stopped
after $6$ trials since we reached convergence of the reliability scores, i.e.,
after trial $6$ there was little change in reliability scores.

From the results we can see that the Dynamic Input Frame configurations
outperform the static ones, both in terms of game score and agreement with EM.
Moreover, the configurations with the Continuous Reliability system
outperformed MV, both in terms of scores and similarity with respect to EM.
For the Dynamic Input Frame with Continuous Reliability System (the best
configuration), the agreement with EM reached a value of $99.37\%$ compared to
$94.40\%$ if the system is replaced with majority voting ($5.26\%$ relative
improvement). In Figure 6, the Dynamic Input Frame with Continuous Reliability
System outperforms all other configurations already after $1$ trial. This
means that the system quickly and correctly learns which players are more
reliable.

Figure 7: 14 AIs reliability training with the Dynamic Input Frame

Figure 7 shows the process of the reliability training with the Dynamic Input
Frame configuration. The reliability scores ($y$-axes) are plotted for each of
the analyzed input frames ($x$-axis). The process is chaotic at the beginning
but stabilizes after circa $100$ frames. After $100$ frames, the estimated
reliabilities correlate with the AIs skills: the top reliability score (red
line) belongs to the Perfect AI. The inputs issued by the perfect AI will be
weighted as $7$ players. The following $3$ reliabilities scores belong to the
$3$ Good AIs. At the end of the experiment, the Perfect AI with the help of
just one of the Good AIs can outvote the rest $12$ AIs combined. Indeed, the
top $4$ AIs alone have an impact of over $18$ players. This exceeds the total
number of players ($14$) because the rest of the reliabilities are less than
$0$. We can also observe the effect of the lower bound $\omega$, which
prevents bad players from disrupting the weighted voting system.

###  V-C Online Experiment with Real Players

In this experimental scenario, we evaluate the proposed Crowdsourcing
Controller with real users. We tested only $3$ configurations out of $4$:
Dynamic Input Frame with MV, Dynamic Input Frame with Continuous Reliability
System, and static frame with Continuous Reliability System. We did not test
the static frame with MV because: (1) the experiment with $4$ configurations
was too long and players’ fatigue could affect the quality of the collected
data; (2) the offline experiments show that this is the worst performing
configuration (see § V-B)

We enrolled a total of $9$ players, all of them were university students with
a scientific background. Before starting the experiment, participants were
trained by playing individually for $3$ minutes. Afterward they had $3$ trials
to achieve their highest scores. Then we Started the multiplayer phase. Each
configuration was played for $3$ minutes for training, followed by $5$ trials
to beat the highest score. The whole process was repeated $2$ times to collect
more data. Exact user instructions can be found in our code repository. During
the experiments the reliability scores were not shown to the players. This was
done to avoid the possibility of the players feeling discouraged due to low
impact of their inputs. There was a short break between the sets of
configurations to give the players a chance to rest.

Figure 8: User scores during first set of trials TABLE III: User studies result summary Name |  | Aggrement
---
w/ EM predictions
| Mean
---
score
| High
---
score
Dynamic Reliability | 0.9619 | 255 | 541
Dynamic Majority | 1.00 | 194 | 516
Static Reliability | 0.9108 | 76 | 434

The game scores achieved by the players during the first set of trials (both
training and game) are shown in Figure 8. The results for the second trial are
comparable. Table III reports the results aggregated over all trials.

Online experiments corroborate the observations from offline experiments with
respect to input frames. The Dynamic Input frame configurations outperform the
static input frame configuration (green line in Figure 8). This is true both
in terms of game score and agreement with EM. Second, in terms of game score,
the Dynamic Input Frame with Continuous Reliability outperforms the other $2$
configurations.

Online experiments differ from offline experiment with respect to the
agreement with EM. We can observe that using a real-time MV system with
dynamic frames yields the same results as using an offline-trained EM. This
suggests that the majority of the players are similarly skilled, thus using a
weighted MV instead of the uniform MV does not make a significant difference.
We further analyzed the agreement among the players which showed that a
particular majority (specifically $5$ players out of $9$) agreed with each
other on issuing the same input over $80\%$ of the times. This can be seen
also in Figure 9, where the reliability scores of the top $5$ players are
clearly separated from the rest of the group.

Figure 9: Player reliability training in one of the sessions with the Dynamic
Input Frame

Even if all the players are similarly skilled, the better performance of the
Continuous Reliability system in terms of game scores might come from the fact
that fewer players with stronger influence more quickly surpass the needed
threshold of votes required by the Dynamic Input Frame, which helps with
beating obstacles that require fast decisions.

At the end of the user studies, participants filled a short survey. Among
different observations, over half of the participants mentioned, in different
words, that they focused more during multiplayer than individual sessions,
which is aligned with previous results about gamification approaches to
encourage crowd workers (see Section II-A). Some participants mentioned the
social aspect of playing together, saying that winning and losing together was
more fun. This suggests that the system could be used as a gameplay mechanic
in EMGs. Some players, however, reported feeling frustrated when they lost
despite seemingly issuing correct inputs, sometimes blaming it on the network
delay or software errors. This implies that players could benefit from even
more visual feedback regarding the decision making of the Crowdsourcing
Controller.

###  V-D Discussion

Next, we summarize the main conclusions and observations from offline and
online experiments. The Dynamic Input Frame proves to be more efficient than a
static approach proposed earlier in the literature. It is able to achieve much
higher scores thanks to more precise predictions of the correct time that the
input should be issued at.

The Continuous Reliability System increases the game score to a large extent
for offline experiment, but to a smaller extent for online experiments. The
reason for this is the observed lack of differences in the skills of real
players. Although a small number of them did in fact perform worse than
others, the majority usually agreed on the input type and time. In this
situation, it is expected that MV gives similar results as the Continuous
Reliability System. Moreover, analyzing players’ inputs proves that
participants usually agreed on the type of the input, with the differences
mostly showing in the precise time of issuing it. The reason for this comes
from the nature of the game that was used to test the system. The real
challenge was in fact issuing the input at the right time, since choosing the
right input was relatively simple as different obstacles were easily
distinguishable.

Both user surveys and observations during the study showed a significant
social aspect of playing the game together, where synchronization seemed like
one of the goals that the players wanted to achieve, knowing it would raise
their chances of getting high scores.

##  VI Conclusions and Future Work

In this paper we present a prototype of an online video game, where multiple
players control simultaneously the same character. We propose the
Crowdsourcing Controller, i.e., a system that: (1) groups real-time inputs
from multiple players into time frames; (2) aggregates the inputs within a
frame and returns a single output command. We propose the Dynamic Input Frame
method, which adjusts the frame length dynamically based on the inputs
received. To aggregate the inputs, we propose the Continuous Reliability
System, which computes a weighted MV by estimating players’ reliabilities.
Offline and online experiments show that the Dynamic Input Frame is better
than state-of-the-art static frames and that the Continuous Reliability System
leads to higher game scores.

As future work we plan to test the proposed crowdsourcing system with a larger
and more diverse set of players. Moreover, we will explore how the proposed
systems cope in different settings, for example scenarios featuring more
agency and creativity on the players’ side. Finally, we want to inspect the
possibilities of automatically optimizing some of the system parameters that
were arbitrarily chosen, for example the TTL of the Dynamic Input Frame.

## References

  * [1] Ahn, L. & Dabbish, L. Labeling Images with a Computer Game. Proceedings Of The SIGCHI Conference On Human Factors In Computing Systems. pp. 319-326 (2004), https://doi.org/10.1145/985692.985733
  * [2] Anderson, S. Watching people is not a game: Interactive online corporeality, Twitch.tv and videogame streams. Game Studies. 17 (2017,7)
  * [3] Brabham, D. Crowdsourcing. (The MIT Press,2013)
  * [4] Dawid, P., Skene, A., Dawidt, A. & Skene, A. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied Statistics. pp. 20-28 (1979)
  * [5] D’Eon, G., Goh, J., Larson, K. & Law, E. Paying Crowd Workers for Collaborative Work. Proc. ACM Hum.-Comput. Interact.. 3 (2019,11), https://doi.org/10.1145/3359227
  * [6] Howe, J. The Rise of Crowdsourcing. Wired. (2006,6), https://www.wired.com/2006/06/crowds/
  * [7] Kim, J., Sterman, S., Cohen, A. & Bernstein, M. Mechanical Novel: Crowdsourcing Complex Work through Reflection and Revision. (2016,11)
  * [8] Lasecki, W., Murray, K., White, S., Miller, R. & Bigham, J. Real-Time Crowd Control of Existing Interfaces. Proceedings Of The 24th Annual ACM Symposium On User Interface Software And Technology. pp. 23-32 (2011), https://doi.org/10.1145/2047196.2047200
  * [9] Lasecki, W. & Bigham, J. Interactive Crowds: Real-Time Crowdsourcing and Crowd Agents. Handbook Of Human Computation. pp. 509-521 (2013,11)
  * [10] Silberman, M., Tomlinson, B., LaPlante, R., Ross, J., Irani, L. & Zaldivar, A. Responsible Research with Crowds: Pay Crowdworkers at Least Minimum Wage. Commun. ACM. 61, 39-41 (2018,2), https://doi.org/10.1145/3180492
  * [11] Stol, K. & Fitzgerald, B. Two’s Company, Three’s a Crowd: A Case Study of Crowdsourcing Software Development. (2014,5)
  * [12] Tomaselli, F., Sanchez, O. & Brown, S. How to Engage Users through Gamification: The Prevalent Effects of Playing and Mastering over Competing. (2015,12)
  * [13] Welinder, P. & Perona, P. Online crowdsourcing: Rating annotators and obtaining cost-effective labels. 2010 IEEE Computer Society Conference On Computer Vision And Pattern Recognition - Workshops. pp. 25-32 (2010)
  * [14] Wodarczyk, S. & Von Mammen, S. Emergent Multiplayer Games. 2020 IEEE Conference On Games (CoG). pp. 33-40 (2020)
  * [15] Ye, T., You, S. & Robert Jr, L. When does more money work? Examining the role of perceived fairness in pay on the performance quality of crowdworkers. Proceedings Of The International AAAI Conference On Web And Social Media. 11, 327-336 (2017)