# All one needs to know about shared micromobility simulation: a complete
survey

Yixuan Liu, Yuhan Tang, and Yati Liu

###### Abstract

As the shared micromobility becomes a part of our daily life and environment,
we expect the number of low-speed modes for first-and-last mile trips to grow
rapidly. The shared micomobility is expected to serve billions of humans,
bringing us considerable advantages. With this growth, shared micromobility
simulation such as docked stations based shared bikes, dockless shared bikes
and e-scooters, are regarded as promising solutions to deal with a large
number of first-and-last mile trips. In this paper, we first provide a
comprehensive overview of shared micromobility simulation and its related
validation metrics. Next, we classify the research topics of shared
micromobility simulation, summarize, and classify the existing works. Finally,
challenges and future directions are provided for further research.

##  I Introduction

Micromobility - a revolutionary transportation strategy - is defined as tiny,
light and even portable vehicles running at a speed below 15 mph. It is now
regarded as a promising way to solve the “last mile” problem. Despite the
existing solution like walking to transit and driving to transit,
micromobility is a less tiring and more economical-friendly alternative.
Therefore, micromobility is placed high hopes for reducing the reliance on
cars for traveling less than 10 miles. They are easy to ride, require less
space, cause almost no pollution and perform well in short-distance house-to-
house transportation. If developed well, micromobility will even cause
voluntary behavioral change [1] and innovate the whole transportation system.

Although there are already a lot of researches about physical road [1],
driving [2] [3] and aviation [4], the researches on aviation are still rare.
Everything new is the unity of forward and tortuous. Competing for road space
with other means of transportation, micromobility makes transportation systems
much more complex. We can’t simply classify them as pedestrians, bicycles or
cars. What changes may happen when micromobility – a new mode of
transportation – is added? Does it affect people’s mode choice or route
choice? Is it more energy-efficient? The previous way of simulation is no
longer suitable for micromobility. What can be called an innovative mesoscopic
micromobility simulation? Therefore, in order to have a better understanding
on micromobility and give full play to its strengths, a systematic
micromobility stimulation is necessary.

Researches on the simulation of micromobility range from microscopic to
macroscopic and increasingly attention is paid to human behavior and decision-
making. Wu J L C[2] is one of the first researchers to simulate micromobility
in 1996 using the Sun Microsystems Mobile IP model. Thanks to the invention of
the Multi-Agent Transport Simulation Toolkit (MATSim), researches on the
simulation of micromobility flourish. To test whether micromobility is
feasible and safe enough to be a new kind of transportation, researchers
concentrate on microscopic simulation targeting one single vehicle. Alberto
Fernández, et al. establish a microscopic agent-based model, where different
types of users can be defined and adjust according to private objectives, to
test different station capacities, station distributions, and balancing
strategies.[3] After the wide spread of micromobility in San Francisco Bay
area, the focus sfhift to mesoscopic simulation to find out passengers’ choice
on a larger scale. Focusing on the impacts of different policies of
incentives, Alberto L´opez Santiago, et al. build and validate the model.[4]
In order to evaluate and compare different alternatives for the system designs
Francesc Soriguera, et al. include rebalancing/reposition dynamics and battery
constraints for e-bikes in their simulation.[5] In addition to the models
mentioned above, previous research contributions also summarize the potential
of micromobility in literature reviews. Abduljabbar R L, et al. consolidates
knowledge on how micro-mobility meet sustainability outcomes in urban
environments through their review.[6] In summary, the research emphasis of
micromobility is turning from traffic flow to people’s behavior.

Different from the papers mentioned above, the contribution of this paper is
three-fold: (1) We propose an exhaustive taxonomy of concepts in the area of
micromobility simulation, and present a comprehensive survey on micromobility
simulation. (2) We provide a comprehensive review on micromobility simulation,
how it is defined, and how it is related to or different from other similar
simulation paradigms, such as transit simulation, ride and hail simulation,
car simulation and pedestrian simulation. (3) We have compiled a list of
challenges and future directions for research in micromobility simulation.

##  II Methodology

###  II-A Taxonomy of micromobility simulation

*Taxonomy for inputs

Inputs are the files or data that are put in, taken in, and operated on before
the micromobility simulation, which determines the basic framework of the
simulation model as well as part of operational rules. It should be noted that
inputs are not limited to variables, files in any formats or even independent
variables can both be inputs, for the import of which is a prerequisite for
model operation. For micro-mobility simulation, basic required inputs are:
configuration files, building the data structure and containing optional
parameters to formulate the operational rules; network files, describing the
features of the transportation network; environmental files, taking into
account the geographical factors(i.e. terrain, elevation, climate, etc) and
social factors(such as population, occupation, land use, communities, etc);
travel demand files, stimulating and determining the features of travel;
behavioral rules files, formulating the travel routes, directions and options.

* Taxonomy for parameters

Parameters decide both the data structure and system structure for
micromobility simulation, and formulate the rules to limit the way that the
elements in models can be done. All the operation and behaviors have to work
within the parameters that had already been set. According to the parameter
setting purposes and the roles of parameters played in the simulation, six
categories of parameters can be found. However, intersections exist for the
usage of parameters can be found in every part of the system.

  * •

Parameters for configuration are the parameters what are usually contained in
configuration files as one of the elements of inputs, determining the
structure both for data and system as well as basic operational rules.

  * •

Parameters for service level represent the efficiency and availability of the
model for evaluation the simulation, such as the availability of bikes.

  * •

Parameters for network and route design determine the detailed structure of
the transportation network and route selection formulations.

  * •

Parameters for constraint set constraints for the operation and behaviors,
such as the charging and discharging rules and time worth. The same parameter
may be set in different ways depending on the researches’ considerations.

  * •

Parameters for utility functions take many factors such as time worth and cost
into consideration in order to evaluate the traveling utility.

The term micro-mobility, in the sense of taking short-trips with small
vehicles, is widely believed in related researches to contain three specific
vehicles: bikes, e-bikes and e-scooters. The National Association of City
Transportation Officials defines shared micromobility as: “all shared-use
fleets of small, full or partially human-powered vehicles such as bikes,
e-bikes, and e-scooters.” Bike sharing system including bikes and e-bikes
exists two common modes, station-based mode requires bikes to be rented or
returned on the docking stations while bikes can be locked optionally in the
service-area in dockless mode [5, 6]. The e-bike sharing system and e-scooter
sharing system consist of e-vehicles, batteries powered by lithium ion,
automatic locking rack and a station for rental and charging. Micro-mobility
makes short trips easier and low-cost, allowing people healthy, fast,
emission-free and effective movement. Also, it will bring environmental and
social benefits for alleviating the problems caused by traffic pollution and
congestion. However, the randomness, asymmetry and peculiarity of the micro-
mobility demands in different areas cause operating agencies many problems,
such as the system unbalance(A simulation model for public) or even system
failure.

As a positive solution to these problems and predict the potential usage,
micro-mobility simulation emerges, and the research scope of which can be
classified into three levels, including the microscopic level, the mesoscopic
level and macroscopic level. Compared to the formulation of management policy
representing the macroscopic simulation, microscopic simulation owns much
utility for simulating the behaviors and interplay-performance of individual
commuters, but the size of the research areas are usually limited. Mesoscopic
simulation has a much larger research area, which is usually a metropolitan
region, focusing on users equilibrium and their usage of utility functions.
Therefore, the results of the mesoscopic simulation could be more persuasive
and reliable.

Establishing a system should take many influential things into consideration,
like the system itself(number, location and density of the station, number of
the vehicles, size of the areas…), demand gereration rules, users generation,
rebalacing strategies, and environmental factors such as weather and policy.
Due to the additional elements of the e-vehicles sharing system, many
operational complications should be introduced into the original sharing
system simulation, such as charging rate, battery constraints et al. Shuguang
Ji et al.(2014) developed a Monte Varlo simulation model of e-bike sharing
system to evaluate the performance [7].

Generally, for micro-mobility simulation in the light of framework there
exists two common categories, strategy-based simulation, which focuses on the
mechanisms or elements of the system[-], and agent-based simulation, which
emphasizes the interactions between involved agents whatever passive or
positive, and more consideration towards behavioral rules, influential factors
and conformity with the reality is given. All the elements of the system in
agent-based simulation is established as an operable unit, which can be
adjusted according to the patterns and schemes. The adjustment of one agent
will affect variables and even cause action alternation of other agents, that
is, the decision of one agent will generate an interaction in the simulation
system. MATSIM is a software developed for agent-based transportation
simulation, with which some reproducation of the micro-mobility sharing system
has been realized. Cornelia Hebenstreit Hebenstreit[4] et al. built a dynamic
bike sharing module on MATSIM with the implement of a within day rescheduling
and choice probability to provide improvement to the current system [8].

###  II-B Inputs — What were the inputs for the simulation? What were the
commonalities and differences across different methods? Follow the above
organization.

Before conducting the simulation, models need to be configured such that the
elements in the model as well as the behavioral rules meet the requirement.
This is what the inputs work for. As explained in the taxonomy, inputs are the
files or data that are put in the model to formulate the whole system, which
can be independent variables and files in any formats. The following text
provides an explication of the basic input files with regard to the micro-
mobility simulation models as well as explanation of the encompassed
parameters.

Configuration files providing the configuration options to construct the data
structure as well as model structure. For MATSim, which is a multi-agent and
activity-based simulation framework commonly used for micro-mobility modeling
and conducting adaptive and co-evolutionary algorithms, a config.xml file is
needed to determine how the simulation behaves with a settings list. It
contains all configuration parameters which are divided into groups according
to the modules. For example, the group related to network contains the
parameter inputNetworkFile; the group related to the Controller includes the
parameter firsIteration and lastIteration.

Network file consisting of nodes and links, providing the routes for travelers
to move around. A network.xml file is a necessity in MATSim to describe the
road transportation network by claiming the features of the nodes and links.
For the nodes, it will provide its identity as well as coordinate values of x
and y. For the links, the identity, starting nodes (e.g., from), ending nodes
(e.g., to), length (typically in meters), capacity (number of vehicles that
traverse the link, typically in vehicle per hour), free speed (the maximum
allowed speed of vehicles, typically in meters per second), perm lanes (the
number of available lanes in the direction from the staring nodes and ending
nodes, all the links are uni-directional), and modes of the links (chosen from
car, bike, and taxi).

Other models on transport network need different kind of inputs. The bi-level
programming model used by Miller et al., (1992) [14] contains the basic
parameters for determining the structure of the network: fearture-related
parameters, such as the denoting and number of the arcs, paths, nodes with
their sets; flow-related parameters, such as the flow on a specific path or
arc; cost-related parameters, such as the unit cost of transportation on a
specific path, the cost incurred by the locating firm to ship, and the total
variable cost of production; demand-and-supply-related parameters, such as the
inverse demand(or supply) function and the quantity supplied at a specific
node; and other parameters.

Area files containing all the distribution of agents, stops, racks, routes can
be clearly seen against the area file. For example, MATSim needs an ESRI file
in Shapefile format to provide detailed population demographics and land-use
patterns.

Population.xml, containing the necessary information about the model for
MATSim. Population is the full set of agents, describing the day plans of
agents in a hierarchical structure (as follows) to generate travel demand. The
minimum example contains a list of persons, and each person has a plan with a
score attribute (obtained in the scoring stage after simulation), and each
plan posses a list of activities, legs as well as routes. If the simulation
starts, the link will be automatically assigned to the nearest activity and a
route for a leg will be figure out.

The activities and legs providing the agents’ actions in each plan: activities
has three parameters: the type of the traveling(home, work and others), the
link and the end time(except for the last activity). End time is sometimes not
required because a duration replaces it in some activities which are
automatically generated by routing algorithms[7]. Link will explicate the
location from which the activity is reached by providing the link or an x and
y coordinate value. Link is required in MATSim so Controller will figure out
and use the nearest link to the given point if the link attribute is missing.
Each leg is assigned with a transport mode and probably with a parameter
trav_time to describe the expected travel time for the leg. Leg will be
directly executed after the last activity has ended.

Parameter route is also required for execute simulation, but it can be
different according to the modes. For example, route in car legs will list the
links for agents to move along it in the given traveling direction, while
route in transit legs will store the stop locations and expected transit
services. Automatically calculation on initial routes for initial plans will
be conducted by MATSim without containing them[7]. When the agent is behaving
in the simulation, plans will be selected to conducted while only one plan can
be executed at a time. Changing of plans occurs when replanning stage is going
on.

Population-level data is the features of the communities, such as the
employment rate, income level, age distribution of residents, regional
function (business, entertainment, university area, uptown or others) , which
will influence the transits’ tendency to some extent. This kind of data is
usually obtained from the official census. Lazarus, Jessica et al., (2020)
[11] added the population-level data - tract-level population, job count,
employment rate, age, income, and gender distribution, which come from the
U.S. Census (ACS, 2016; LEHD, 2013)- to the basic trip-level data as one of
the input files.

Travel demand files are needed for all the micro-mobility simulation models.
Generally, census data, observational experiments, and stated-preference
surveys are the common demand data resources. The datasets include the basic
trip information such as the trip duration, start (or end) times and location
(usually the coordinate of latitude and longitude, that is the origin and
destination of the trip), as well as users’ features in some studies. While
some researchers found other resources of data for demand generation, like
transport card transactions. In Lovric et al. (2013) generated the demand by
smart-card transactions (individual check-in and check-out activities with
occurrence time and location) data collected in a Dutch urban area, and
inputted them into the model.

3D Modeling of micro-mobility vehicles Deepak Talwar[10] modified the LGSVL
Automotive Simulator to include micro-mobility vehicles by adding the 3D
modeling of micro-mobility vehicles and extra components into the simulation
environment. According to the popularity in urban society and appearance as
well as motion models, five different kinds of micro-mobility vehicles were
chosen to be simulated.

*summarize

In short, all the variables, data and files which are put into the model for
formulating the basic rules and framework can be collectively referred to as
inputs. Basically inputs for model configuration, network designing, setting
up the impact environment, demand generating and behavioral rules are
required, while detailed required inputs will vary according to the applied
methodologies.

###  II-C Variables

Traffic flow of micromobility, like any other traffic flow in the real world,
is continuously changing and evolving. Actually, the evolution of road traffic
is even more complex thanks to the interaction of various means of
transportation. It’s impossible to restore the whole traffic system
completely. Because there’re endless variables and the things will be
different if any single variable changes. Simulating large road networks and
enormous agents will be a nightmare when running time is taken into account.
So finite representative variables must be chosen to reflect the real traffic
as well as minimize the uncertainty and inaccuracy caused by ignoring the
inconspicuous features of road traffic.

To find out the similarity of the variables, we first categorize them into the
following six groups. Each of them contains some examples. On the other hand,
as explained in taxonomy, parameters contained in the whole micro-mobility
simulation can be generally divided into five categories according to the role
played in the simulation.

####  II-C1 Related objectives

Then, we focus on different factor types and typical examples correspondingly
to answer the following 4 questions:

Why are these factors chosen?

How are these factors related to their simulations?

How can these factors reflect the evolution of the model?

What principal should we follow to adopt these factors into our model?

##### Variables related to a person or a passenger

Subjective needs are one of the key elements to influence and even alter one’s
decision, thus leading to a series of changes to the transportation system.
Therefore, it’s necessary to visualize and quantify people’s needs and put
them into simulations as factors.

Wang I L, et al. [1] simulate the service requests by setting the lower bounds
of bike inventory as and the upper bounds as in order to find proper bike
redistribution strategy. If the inventory level isn’t within the bounds (i.e.
), a service request will be assigned. Only one service request is considered
at a time. If there’s a loading request (i.e. ), then bikes should be unloaded
from vehicle to the site. Similarly, bikes should be loaded from the site to
vehicle. and represent a safety bike buffer or empty rack buffer respectively.

If more detailed information is considered, factors reflecting the trip and
the access and excess condition should be included. For instance, travel and
connect distance and total bicycle access and egress time are used to
represent the travel demand by Lu M, et al. [5] and Romero J P, et al. [7]
respectively, because (i) they are most likely to influence passengers’ mode
choice; (ii) with the distance data, the interaction between travelers can be
simulated.

Some other factors should be included for specific researches. Bicycle access
time worth is considered by Romero J P, et al. [7] in order to meet their
objective to take passengers’ personal time value into account and then to
achieve total efficiency and minimal social cost. Elhenawy M, et al. [8]
regard the Home Index (HI) - the percentage of agents that finish their trips
at home location – as one of the two criterions to evaluate the behavior of
their agent-based model when the central agent rules are changing, so they
introduce the home location as a factor to finally obtain the HI.

In conclusion, the PR factor in an agent-based simulation targeting MM should
show individual’s travel demands. The demands can be general or specific
according to different objectives. Factors about the trip itself and its
access and excess condition are widely used.

##### Variables related to vehicles

In agent-based simulations, vehicles and people are equally important. In most
situations, they are both chosen as agents. On the other hands, vehicles have
different features compared to people and MM is also unique to traditional
vehicles like cars. In this part, we discuss how MM act as the role – vehicle
– in transportation system and how to reflect them through factors.

Firstly, traffic flow data. Elhenawy M, et al. [8] record the current location
of each vehicle every clock tick to check if its current location coincide
with the origin of the trip to find if the vehicles (e-scooters in the paper)
can fulfill the demand request. To reflect the evolution of the simulation,
the demand generator gives all e-scooters new demand requests including
information related to trips like location.

Secondly, factors about the vehicles itself. E-bike battery range [2][8]
reflects the endurance mileage of the vehicle and it’s crucial when allocating
orders. When the e-bike is in used, the battery level will go down and when
it’s in a station, it can be recharged. The inclusion of battery level and
other similar factors that shows the feature of vehicles can affect people’s
choice to some extent and thus make the simulation more accurate. For
instance, users may favor bikes with higher battery level. Bikes with battery
levels under a certain percentage will be showed unavailable.

Thirdly, factors that shows how the vehicles interact with other elements,
such as the inventory level, the current request [1], the cost of travel [5],
travel time [7], the number of available bikes [9]. Wang I L, et al. [1] check
if the inventory level is within the bounds to decide if they can accept the
service request or not. Additionally, they also record the ID of the service
request that the vehicle is serving currently in their simulation to better
manage them. Balać M, choose adoption rate as a factor corresponding to their
long-term simulation. Changing of the adoption rate alter some outputs like
access and egress time and rejection rate. Lu M, et al. [5] set the cost of
travel as a factor since most MM companies use tiered price, so the travel
bill vary from person to person and trip to trip. Whether the price is
competitive enough is crucial to the survival of MM. Romero J P, et al. [7]
include travel time to solve the planning problem to achieve the smallest
social cost. Caggiani L, et al. use the number of available bikes reflect the
situation of each station.

Fourthly, abstract factors that are used to adjust the model to make it more
close to the reality. In MATSim’s official guide [4], flow capacity factor and
storage capacity factor are included. They should be set corresponding to
different scenarios.

In conclusion, VH factors should be carefully selected and set for a
satisfying simulation. Traffic flow data and factors about the vehicles itself
are the basis since they reflect the role of MM in transportation system.
Factors showing how the vehicles interact with other elements are also
important when considering the evolution of the model. Abstract factors may
sometimes be introduced to adjustment.

##### Variables related to roads

Road condition matters a lot in travel mode choice, efficiency and
comfortableness, etc. Chen L, et al. include traffic events as a quantitative
index about road condition. They record each type of traffic event on its
nearby stations in the next hour after the traffic event occurs. These events
may affect people’s choice and the vehicles’ running route.

Though road condition seems as an important factor, not many scholars take it
into account. Some neglect it as they focus on macroscopic simulation and road
condition is too micro for them. For those who do microscopic and mesoscopic
simulation, perhaps they finally give up considering road condition because it
takes more simulation time and compacity or they haven’t found a good way to
reflect road condition. If proper factors can be found or more powerful
simulation engine is introduced, it’s still necessary to take RD factors into
account.

##### Variables related to context or surroundings

Context and surroundings can easily affect people’s decision on
transportation. One may not choose micromobiliy when it’s raining or it’s too
hot. Chen L, et al. [6] select four factors related to context or surroundings
– date and time, weather condition, air temperature and social event. For the
factor date and time, they first spilt two day types – weekdays and weekends.
Then, weekdays are divided into 4 parts - morning rush hours, day hours,
evening rush hours and night hours; weekends are divided into day hours and
night hours. Thanks to date and time factor, various bike usage pattern
according to the time of the day can be observed and studied. As to weather
condition, they define the following five groups: clear, cloudy, rain, snow,
and haze. When it comes to air temperature, they split into below zero, cold,
comfortable, and warm. 0∘C, 10∘C and 22∘C are set as boundaries.

##### Variables related to the condition of the origin and destination point

One may choose an origin point (O point) when there’s enough bikes and the
road around is bicycle-friendly; similarly, one may finish their trip (D
point) in a station with an empty space or a place near a subway station.
Therefore, the condition of the OD point is a factor that can’t be ignored.
Some factors are related to both O point and D point (CO & CD), while others
only affect one point (CO or CD).

Most CO/CD factors are CO & CD factors. Wang I L, et al. [1] introduce the
full rate of a rental site and categorize the sites into different levels
according to it. However, the full rate of a rental site sometimes act
differently in O points and D points. =50% or 100% is almost the same for an O
point, but means it can’t be a D point. In turn, =0% or 50% is almost the same
for an D point, but means it can’t be an O point. So, should be distinguished
according to different points though it’s the same factor. Soriguera F, et al.
[2] also care about whether the rental site is full, but they use it in a
larger scale. They only care about full stations (marked 1) and empty stations
(marked 0). Then the percentage of full and empty stations at a given instant
is included in their simulation to get the peak value and low value of vehicle
using. Balać M, et al. [3] use a figure to show the location of all the pickup
and dropoff stations. Lu M, et al. [5] include cost of connect mode to get a
more accurate travel cost. It varies as time goes by due to the changing
external environment. Romero J P, et al. [7] include number of docking
stations and per-station cost to calculate the minimum social cost. Elhenawy
M, et al. [8] add the availability of e-bike to the four state variables. The
e-bike will be shown unavailable if someone else is using it or it has been
rented.

Some factors are only related to D points. In Wang I L’s simulation, [1] a
better bike-return site, different from the preferred one, is specified by a
commuter. If the bike-return site is expected to be full, then return the
passenger will be suggested to return his bike at a nearest adjacent sites.
Caggiani L, et al. [9] turn to the number of docking slots (free or full) as
variables in their bike sharing system simulator to mark the whether the D
point is available.

####  II-C2 The role in the model

##### Configuration

Configuration parameters work for building the data structure, model framework
as well as simulate the environmental factors of the system. As described in
the Inputs part, configuration parameters will be defined and explained in the
inputs files. Further subdivision yields two more categories: parameters for
framework and environment.

Parameters for framework configuration build the structure of the system or
model, ensuring simulation to behave with a settings list. For MATSim, they
are usually pairs of a parameter name and a parameter value, determining the
data structure and model infrastructure. And the settings of these parameters
depends on the features of data sets used in studies. Some of them are
selected from a variety of patterns, while some are assigned specific values.

Parameters for environment endow the model with more details to promote the
model has more realistic environment, and finally to gain more reliable
results and solutions. Several aspects including infrastructure, social
development, geographical elements and others could be taken into account. For
infrastructure, number and locations of bicycle infrastructure(stations,
lanes, etc)and other public infrastructure(parks, recreational area,
commercial venues, industrial region, metro stations, bus stops, etc)are
determined according the chosen cases. For social development, resident
income, density and distribution of population, community and others can be
taken into account. Temperature, elevation, slope, time period(morning peak
period and evening peak period are commonly chosen), weekdays or weekends,
weather, seasons and others are commonly considered geographical parameters.
The specific settings depend on the cases.

##### Service level

Some parameters are considered in many studies to measure the efficiency and
availability of the model, that is, customer service level. Coverage level,
bicycle availability(i.e. existing available bikes in the station), bicycle
stations availability(i.e. existing available bicycle docking for users to
lock), maximum walking time or distance, traveling cost are mainly used
measuring parameters to conduct behavior rules, estimate the simulation
results and even optimize the facility distributions in cases.

Likelihood of finding a bike/parking works when agents verify their next trip-
activity by searching for available bikes at the origin station or docking
racks in the destination station within the maximal radius. Cornelia
Hebenstreita et al.[1] proposed a utility function to gain the probability,
which includes three vital parameters: the intensity and direction of the
influence quantity at alternative mode, the observed influence quantity of
alternative mode for all agents, and the unobserved error term.

A similar parameter defined by Lin J-R et al.,[16] was named “availability” to
measure the ratio of number of public bicycles available to station capacity.
Therefore, the value of availability ranged from 0 to 1(from empty to full),
and the stations were divided into five categories, among which the category
with the lowest availability and the highest availability will be taken into
account for rebalancing activities.

Max walking distance or Max walking time is taken as a vital parameter by
Miller et al.(1992) [14]. The value of walking from one node to at least one
zone will be calculated to be assured to less than the given max walking time,
for guaranteeing the walking worthwhile to substantially exist.

Maximal distance is used to judge the agent choice by calculating the
distances between origin and the nearest station and between destination and
the nearest station for determining the start and end station. Cornelia
Hebenstreita et al.[1] generated a preprocessed bike-sharing plan, and
therefore the sequence transportation mode (walking, riding and taking a bus)
will be decided according to the given distance limitation (maximal distance,
access distance, and egress distance) and other parameters.

##### Network and trips

As described in the taxonomy above, these parameters act in setting up the
transport network, trip generating and distribution. Multiple ways have been
applied to address the network design problem by using different variables and
assumptions. Commonly, the set of intersections and roads are denoted as vital
elements of network model. Related constraints are denoted by parameters which
are explicated in Part 2.3.2.4 (Constraint). For example, Duthie J et al.[17]
formulated models which connect origin-destination(OD) pairs with bicycle
paths while taking the bicycling service level and total cost into
consideration at the same time. Another solution is Bi-level mathematical
programming model, and at the lower level of which benefits to cars and
bicycles as well as parameters related to travelers’ behaviors or preferences
will be considered for better determination. Romero, J. P. et al.[15] used a
genetic algorithm to search for the distribution of a given number of docking
stations, in order to maximize the number of users. Therefore, parameters
contained in the iterative modal split and network assignment model, as well
as other additional parameters, such as the network-related variables(bicycle
lanes, number and locations of bicycle docking stations).

Four-step transportation forecasting model(or urban transportation planning
model, that is, UTP model) is commonly used to estimate the number of vehicles
or people that will use a specific transportation facility. For example, the
number of the vehicles on a planned road, the number of people visiting the
specific bike stations to find an available bike, or the number of people
reaching the destination station for parking. Firstly implemented in the 1950s
at the Detroit Metropolitan Area Traffic Study and Chicago Area Transportation
Study (CATS), the four steps are: trip generation, trip distribution, mode
choice, and route assignment.

In trip generation step, trip purpose, land uses, household demographics, and
other socioeconomic parameters are set for determining the frequency and
locations of origins or destinations of trips.

In trip distribution step, origins and destinations generated in the last step
are matched in this step. Generally there are two basic methods: growth factor
methods and synthetic methods[13]. Constant factor method, average factor
method, Fratar method and Furness method are typical growth factor methods,
which are all older methods to conduct trip distribution work. Commonly used
synthetic methods are gravity model (that is, an entropy maximizing model) and
destination choice models. Gravity model applies to mono-centric urban regions
with little attention on accessibility to transit, therefore it’s not
appropriate for many urban cases which has many dominant attraction regions
and various types of transits such as intown-to-surburban, intown-to-intown
and surburban-to-surburban trip flows. However, destination choice models
overcome these limitations by applying proper specifications of utility to
assure logically responding to changes in levels of service and demand. The
following table describes the parameters consideration in the models:

##### Constraint

Time worth is the traveling cost per unit of time. Most of the simulation
model try to minimize the total traveling cost, while each travel cost could
be obtained by multiplying the time worth factor and the time. Miller et
al.[14] set three time worth parameters: bicycle travel time worth, bicycle
access time worth, and car travel time worth. While Parameters related to
travelers’ behaviors or preferences will be considered in the lower level for
better determination.

Cost is a vital factor which is considered in many ways for micro-mobility
simulation. It’s commonly taken account in the network design model, route
choice model, traveler behavioral rules, utility functions, etc. Cornelia
Hebenstreita et al.[1] took the marginal cost of traveling into consideration,
which consists of time dependent costs, route determined costs and monetary
costs. Denoted by Miller et al.[14], social cost consists of three parts:
bicycle travel costs, total station cost and car travel cost. Given that the
inputted parameter time worth(e.g. the traveling cost per unit of time) is
determined, the values of the three cost parameters could be obtained and
changed according to the results from the simulation. For instance, bicycle
travel cost is the sum of the product of total bicycle travel time and bicycle
travel time worth as well as the product of total bicycle access and egress
time and bicycle access time worth. At the upper level of Bi-level programming
model, cost(including social, economic, environment, etc) is considered to
optimize the decision.

Charging/discharging rules should be considered in e-bikes simulation
framework. In most of the literature, a simplified linear loss is generally
used as the discharging energy flow curve, because the various influential
aspects (battery capacity, riding speed, weight of rider, weather, gradients
and others) in reality could complicate the calculation. For the charging
rule, some of the studies will still use a linear curve while others proposed
their functions to depict the changing of the energy.

Minimal distance can be the given distance between one node to the nearest
docking station in the study of Miller et al.(1992) [14], who didn’t consider
the cases that have docking stations closer than the minimal distance, in
order to further make a reduction on the search space. In the reference [14],
another minimal distance was also proposed to restrict that location of
docking stations that any two of them cannot be closer than the minimal
distance.

Population size were set differently in the outer Genetic Algorithm with the
same number of docking stations to find the maximal number of public bicycle
users[14].

Agent memory size is set to constrain the number of plans, and only the plans
with the highest probability will be kept if the limit is exceeded to generate
improving correlations. This parameter will act in the replanning process.
Dubernet et al.(208;15)[4]

User group provides the attitudes for the aspects in the riding, such as the
routing choice(safety, smooth, gradient, comfort, congestion, environment,
other). In MATSim, a user group and a bicycle type will be assigned to an
agent. Cornelia Hebenstreita et al.[1] evaluate for all attribute groups from
0-10 to develop the parameter named bike-friendness, which can be used to
generate the parameter called personal weighting for choosing the best route.

##### Utility

Utility functions are used to evaluate the performance of operation, while
parameters and factors concerned in current studies are different. Standard
MATSim utility function[3](also named MATSIM scoring function) can be used to
determine whether the equilibrium is reached. Agents will gain utility when
they are performing activities and dis-utility when they are traveling. It was
firstly developed by Charypar and Nagel(2005) on the basis of the Vickrey
model for road congestion, contains the definition of the activity utility and
travel utility. The basic function considers activity duration, waiting time,
as well as penalties of late arrival, not staying long enough and very short
trip. All the parameters included in the basic function are explained in Table
x.

Table x

|Table 3:|||parameter ||explanation ||parameter ||explanation || ||the utility
of an activity || ||the performed activity duration || ||travel dis-utility
for a leg || ||related to the marginal utility of activity || ||||||the
duration when utility starts to be positive || ||the waiting time spent ||
||the direct marginal utility of time spent waiting || ||the late arrival
penalty || ||the waiting time || ||the penalty for not staying long enough ||
||the activity starting time || ||the penalty for a very short activity ||
||the latest possible penalty-free activity starting time || ||the activity
ending time || ||the earliest possible activity end time || ||related to the
penalty for not staying long enough || ||the shortest possible activity
duration || ||a mode-specific constant || ||the direct maginal utility of time
spent traveling by mode || ||the travel time between two activity locations ||
||the marginal utility of money ||$\Delta$ ||the change in monetary budget ||
||the marginal utility of distance || ||the mode-specific monetary distance
rate || ||the distance traveled between activity || ||public transport
transfer penalties || ||a 0/1 variable signaling whether a transfer occurred
||

Although the original purpose of the scoring function was to choose departure
time, further choice dimensions have been verified to successfully realize. In
order to conduct more reasonable simulation, many properties of riding were
taken into account to extend the standard function. Dominik Ziemkea et al.[2],
who firstly proposed the concept of person dependent perception of trip
duration[1], added an infrastructure and a comfort component to revise the
travel utility, considering the cycling infrastructure, pavement conditions
and riding smoothness. Felt travel time (as) was then introduced by Cornelia
Hebenstreita et al.[1] acting as one of the scoring for riding to deal with
different extent of preference among factors (route, comfort and so on). It
has two sub-parameters, that are the felt travel times of each edge and the
marginal costs of traveling. In addition, some studies conducted the
integration of social networks in multi-agent simulation of MATSim framework.
Hackney et al.[5] proposed their formulation of socially-networked utility
specifications, that joint utility function allows the attributes, utility as
well as attitudes of at least one agent to act as decision variables in the
utility of another agents. Therefore, they attached attention to social
relationships, and four social interaction functions were explicated. Instead
of only focusing on intrahousehold ties, Dubernet et al.[4] designed a model
containing explicit coordination of individuals to represent joint decisions,
which can deal with complicated network topology. They added two parameters to
the basic function, a random utility , for each activity to express the
preference variations; and the other, the utility of time passed with sicial
contact.

*Summary

On the whole, five kinds of parameters involved in micro-mobility simulation
can be found. Configuration parameters contained in the inputs will determine
the data structure and framework of the model; service level-related
parameters service for representing the travel indicators of the system as a
judgment of whether the system runs well; parameters in network and route
designing process construct the structure of the transportation network and
formulate the route selection behavior rules; parameters for constraints will
set the limitation of some of the values or assigning quantifiable values to
an otherwise unquantifiable parameter; parameters works in utility functions
will contribute to evaluating utility and performance of the system. What
calls for special attention is that parameters are the basic elements of the
simulation, and therefore are partly involved by inputs. Some of the other
parameters do not need to input, for the implementation of the simulation will
assign them values.

###  II-D Calibration methods — What were the methods that were applied for
calibration? How were they different from each other? How were their usability
related to the inputs or parameter settings? Follow the above organization

Compared with traditional macroscopic models, microscopic models are more
commonly used to study traffic flows in smaller areas where calibration is
required due to the increased number of model parameters (Fellendorf &
Vortisch, P, 2001). Numbers of parameters were used to validate against field
data. To find out the similarity of the validation metrics, we divided them
into two categories: traffic flow characteristics and passenger behavior.

TABLE I:  Table x: The classification of validation metrics
---
Metrics type |  Abbreviation |  Example(s)
Passenger/driver behavior |  |  standstill longitudinal,
distance between the stopped vehicles,
headway time in seconds,
Contact duration(time two visitors remain in contact),
number of neighbors,
Inter-contact time (ICT) is the time elapsed between starts of two successive
contacts of the same pair of
devices/visitors.
Inter-contact time (ICT) is the time elapsed between starts of two successive
contacts of the same pair of
devices/visitors.
Inter-contact time(the time elapsed between starts of two successive contacts
of the same pair of visitors),
The percentage of new contacts
traffic flow |  VH |  The road network flow capacity,
Transit vehicle capacity/volume,
the unsaturated speed,
Trip mode share,
Tour mode share,
Trip length frequency,
Volume of trips,
Trip distribution,
Queue length,
Total delay time,
Total travel time,
Cost of total delay

  * •

Calibration metrics related to passenger (PR)

  * •

Calibration metrics related to vehicles (VH)

TABLE II:  |  [7] |  [5] |  [9] |
---|---|---|---|---
frequency of no battery |  $\surd$ |  $\surd$ |  $\surd$ |
frequency of no bicycle |  $\surd$ |  $\surd$ |  $\surd$ |
service rate |  $\surd$ |  |  |
margin cost |  |  $\surd$ |  |
felt travel time |  |  $\surd$ |  |
waiting list handling |  |  $\surd$ |  |
station selection |  |  $\surd$ |  |
derivation of bs types |  |  $\surd$ |  |
access time(distance) |  |  |  $\surd$ |
frequency of full stations(can not lock) |  |  |  $\surd$ |
leave for empty stations |  |  |  $\surd$ |

Agent-based (AB) transport models simulate the travel demand at the scale of
individuals. In general, this level of detail requires higher computational
capacity and longer runtimes, compared to aggregated models that simulate
traffic flows. To run large-scale AB scenarios in reasonable times, the use of
small subsamples of the population (down sampling) is common practice [10].
However, down sampling will definitely lead to uncertainty [11]. How to keep
balance between a proper scale and certainty is critical to multi-agent
simulation.

5% of agents and 40-50 iterations are suggested in a scenario where results
are analyzed only at a highly aggregate level [10] [12]. Admittedly, the
requirements of the policy scenarios may still affect the most suitable scale
factor and network density for each application. For detailed analysis of
small areas, the random effects across simulation runs should be also taken
into account [11]. The effects of scaling on public transport trips require
further attention, since vehicle capacities cannot be scaled down in the same
proportion.

So, if the computing power is allowed, we can set the agent rate at 30%, in
order that the occupancy of the public transit aren’t adversely impacted [13].
The suggested rate can be summarized in the following table:

TABLE III:  Suggested agent rate |  Population sample |  Public transit included
---|---|---
5% |  Small (2.9 million total trip per day) |  No
30% |  Large (3.5 million private cars & 1.7 million public vehicles) |  Yes

Additionally, the adverse impact of population sampling is determined mostly
by the patterns of the agents’ behavior rather than by the traffic model. So,
we’d better focus on the patterns of the agents’ behavior.

TABLE IV:  |  sensitive analysis |  linear regression analysis
---|---|---
[7] |  activity duration(trip length) - demand - min bikes(batteries) needed |  Activity duration has a strong effect on the number of e-bikes and batteries needed to serve the demand.
R2=0.83min e-bikes required
R2=0.84min batteries required |  Battery requirements are about twice as sensitive to trip length as e-bike fleet requirements.
a strong positive relationship between the minimum number of e-bikes/batteries required and the trip rate |  A correlation between the minimum number of e-bikes/batteries and the trip length is less evident.

###  II-E Comparison to other models

Compared to Equation-Based Models(EBMs), Agent-Based Models(ABMs) are more
detailed, with higher accuracy [14], and can complement EBMs. ABMs allows
further exploration of macroscopic implications based on lab experiments,
generating new hypotheses, and determining the sensitivity of the results.
Akka (which we decide to apply) is one of important actor(agent) models. It is
feasible, reliable and scalable, and Akka dispatcher can be used to iterate
over multi-scale application modules [15].

TABLE V:  Category |  Performance |  Speed |  Maximum number of nodes cluster |  Model size |  Reference
---|---|---|---|---|---
Control and monitoring of agent-based networks |  The largest gain in performance is about 31% |  250 frame per second(FPS) |  1000 |  2400 cells |  [16]
average CPU utilization is 10% |  15 to 30 seconds to add 20 nodes |  2400 |  |
High efficiency in CPU utilization |  an average of 11 minutes on 144 nodes (3456 cores). |  2232(53,568 cores) |  randomly populated grid of size 5.20 $\times$ 107 |  [17]
Virtualization for control and monitoring |  3% improvement in deduplication rate |  89.76% overall reduction in migration time |  |  83% reduction in the overall storage of images |  [18]
Control and monitoring other networks via akka |  In the 10-iteration case, this slows the job down by 50s (21%) on average. |  the analytics is done over 10 minutes, sensors collect readings every 10 seconds |  8000 |  |  [19]

##  III Impacts/Application Fields

###  III-A Overview of the applications for micromobility simulation (Energy,
system design, system optimization, demand prediction, etc..)

The rising of micro-mobility services over the past decade promotes an
emerging trend in studies and developments regarding design and management
methods. The idea of improving the efficiency of a system is to better
allocate resources following the demand. Hence, existing studies concerning
the operation of a micro-mobility service mainly focus on either the supply-
or the demand-side of the reality. Based on how it manages to cooperate real-
world data with its research purpose, these studies could be categorized
mainly into analytic and simulation-related topics. Analytic studies utilize
data for static pattern analysis through methods such as modeling, Spatio-
temporal analysis, etc. These help to discover potential long-term or periodic
interactions between demand and supply concerning internal (e.g. user socio-
demographics), external (e.g. weather, artificial environment), and trip-
related (e.g. origin, destination, distance, time, and date) factors [20].
However, short-term prediction and analysis are more important for real-time
management. Studies on simulation of micro-mobility services focus on
capturing dynamics of system demand and supply by modeling the behavior of
each element within the system, which is more relevant to the topic of this
paper.

Simulation has been developed and utilized as a tool in the design and
management of transportation infrastructure for decades. Based on the scales,
they could be either microscopic, mesoscopic, or macroscopic simulations.
Well-known tools of which include SUMO [21] , MATSim [22] , and PTV Visum.
However, as an emerging mobility service, simulations of the micro-mobility
system are now getting attention from the research community across the globe.
The simulation normally takes information about O-D or trips of users as
inputs, presets a series of parameters that help depict user behavior (e.g.,
mode choices), and generates outputs for analysis.

Some of the studies in this area built models from a macroscopic perspective,
where models consider users from a large scale area with normally static O-D
demands and route choices. Caggiani et al. [23] first proposed a simulator for
validating an algorithm of reposition. It takes preset pickup demand as inputs
for each discrete time interval, derivesdestination choice based on relative
O-D attractiveness and the nature of the trip, removesusers who fail to hire a
bike, and sets those who fail to return the bikes as waiting. The model was
later used and tested in a subsequent study [24] on proposing a dynamic
simulation-based fleet reposition algorithm. In 2014, Ji et al. [7] evaluated
the efficiency ofthe shared e-bike systems through a proposed Monte Carlo
simulation model. The model consists of a trip generation module and
processing events, which handle demand generation and the arrival, return, and
recharging of e-bikes, respectively. But the paper only considered round
trips, and the trip demands and round trip length were randomly derived from
distributions instead of real-world data. Similar approaches are seen in other
studies [25, 26, 27]. Despite the rather low computing costs and sufficient
precision, macroscopic models fail to depict the behavioral properties of the
elements within a system. Many of the above-mentioned studies utilize local
data to calibrate a distribution, which only applies to the condition of the
data source. Besides, the static O-D demands and route choices can’t describe
situations where users switch to other transport modes during the
trip.Mesoscopic simulation, which takes behaviors and dynamics of elements
into account while keeping the simplicity by clustering different users, has
become a widely adopted alternative in recent years. And agent-based models
(ABM) and multi-agent simulation rise amongst the most popular simulation
approaches. In 2018, Hebenstreit et al. [28] developed two modules for MATSim
agent-based simulation, which enable a simulation of bicycle traffic and bike-
sharing within a multimodal context. The paper wasone of the earliest studies
that consider user choices and modal shifts. Soriguera et al. [29] presented a
comprehensive simulation model based on Matlab to support the planning and
design of a bike-sharing system. Repositioning trucks and electric bikes
alternatives were included in this model. The other two subsequent
studies[13],[14] also explored the potentials of ABMs with different settings
of agents and parameters using data from BiciMAD. In short, the above-
mentioned works on micro-mobility simulation mainly focus on the balancing
problem within micro-mobility systems. The mesoscopic models have higher
complexity for computation, but could better capture dynamics of users
regarding modal splits and route choices during the trips. Nevertheless, most
of the existing studies research on a local area with small amounts of trip
inputs. And they lack a systemic analysis for energy efficiency. Hence, the
proposed simulation model in this paper highlights the energy efficiency of
the micro-mobility system. Incorporated with Akka distributed computing
framework, the proposed method enables fast processing of a mesoscopic model
with concrete trip plans (e.g. home-based trip, work-based trip, non-work-
based trips, etc..) and derives energy and emission outputs for further
analysis

The Behavioral Energy Autonomous Mobility Model (BEAM) is a microscopic agent-
based travel demand simulation framework based on the Multi-Agent Mobility
Simulation Model (MATSim) [30] . Agents are measured by traveling time,
expense or costs and choice preferences. Successor to MATSim, BEAM
incorporates the AKKA library aiming to perform scalable analysis of massive
amounts of data from urban transportation systems, in particular to improve
the analysis of the benefits of plug-in electric vehicles (PEVs) [31] . BEAM
outperforms MATsim in three ways. First, BEAM’s mode choice is exogenously
defined. BEAM allows for multi-modal choices, including walking, biking,
driving alone, riding in a car, and a variation of walking-riding-driving.
Secondly, BEAM’s behavioral focus is intra-day planning, which means that BEAM
can better reflect and simulate modern transportation. Finally, BEAM specifies
a power threshold range constraint for electric vehicles.

## References

  * [1] W. Huang, C. Yan, and X. Jiang, “Chemical and rheology evaluation on the field short-term aging of high content polymer modified asphalt,” Tech. Rep., 2019\.
  * [2] C. Chai, J. Lu, X. Jiang, X. Shi, and Z. Zeng, “An automated machine learning (automl) method for driving distraction detection based on lane-keeping performance,” _arXiv preprint arXiv:2103.08311_ , 2021.
  * [3] J. Everts and X. Jiang, “Making sense of electrical vehicle discussions using sentiment analysis on closely related news and user comments,” _arXiv preprint arXiv:2112.12327_ , 2021.
  * [4] A. Bauranov, S. Parks, X. Jiang, J. Rakas, and M. González, “Quantifying the resilience of the us domestic aviation network during the covid-19 pandemic. front,” _Built Environ_ , vol. 7, p. 642295, 2021.
  * [5] C. Hebenstreit and M. Fellendorf, “A dynamic bike sharing module for agent-based transport simulation, within multimodal context,” _Procedia computer science_ , vol. 130, pp. 65–72, 2018.
  * [6] S. Gössling, “Integrating e-scooters in urban transportation: Problems, policies, and the prospect of system change,” _Transportation Research Part D: Transport and Environment_ , vol. 79, p. 102230, 2020.
  * [7] S. Ji, C. R. Cherry, L. D. Han, and D. A. Jordan, “Electric bike sharing: simulation of user demand and system availability,” _Journal of Cleaner Production_ , vol. 85, pp. 250–257, 2014.
  * [8] C. Hebenstreit and M. Fellendorf, “A dynamic bike sharing module for agent-based transport simulation, within multimodal context,” _Procedia computer science_ , vol. 130, pp. 65–72, 2018.
  * [9] F. Soriguera, V. Casado, and E. Jiménez, “A simulation model for public bike-sharing systems,” _Transportation Research Procedia_ , vol. 33, pp. 139–146, 2018.
  * [10] C. Llorca and R. Moeckel, “Effects of scaling down the population for agent-based traffic simulations,” _Procedia Computer Science_ , vol. 151, pp. 782–787, 2019.
  * [11] M. Paulsen, T. K. Rasmussen, and O. A. Nielsen, “Output variability caused by random seeds in a multi-agent transport simulation model,” _Procedia computer science_ , vol. 130, pp. 850–857, 2018.
  * [12] C. Zhuge, C. Shao, and X. Yang, “Agent-and activity-based large-scale simulation of enroute travel, enroute refuelling and parking behaviours in Beijing, China,” _Journal of Computational Science_ , vol. 38, p. 101046, 2019.
  * [13] A. Saprykin, N. Chokani, and R. S. Abhari, “Uncertainties of Sub-Scaled Supply and Demand in Agent-Based Mobility Simulations with Queuing Traffic Model,” _Networks and Spatial Economics_ , pp. 1–30, 2021.
  * [14] K. H. van Dam, A. Adhitya, R. Srinivasan, and Z. Lukszo, “Nuances of benchmarking agent-based and equation-based models of an oil refinery supply chain,” in _2008 First International Conference on Infrastructure Systems and Services: Building Networks for a Brighter Future (INFRA)_ , 2008, pp. 1–6.
  * [15] R. Wyrzykowski, E. Deelman, J. Dongarra, K. Karczewski, J. Kitowski, and K. Wiatr, _Parallel Processing and Applied Mathematics: 11th International Conference, PPAM 2015, Krakow, Poland, September 6-9, 2015. Revised Selected Papers, Part I_. Springer, 2016, vol. 9573.
  * [16] M. Janczykowski, W. Turek, M. Malawski, and A. Byrski, “Large-scale urban traffic simulation with Scala and high-performance computing system,” _Journal of Computational Science_ , vol. 35, pp. 91–101, 2019.
  * [17] J. Bujas, D. Dworak, W. Turek, and A. Byrski, “High-performance computing framework with desynchronized information propagation for large-scale simulations,” _Journal of Computational Science_ , vol. 32, pp. 70–86, 2019.
  * [18] N. M. Tyj and G. Vadivu, “Adaptive deduplication of virtual machine images using AKKA stream to accelerate live migration process in cloud environment,” _Journal of Cloud Computing_ , vol. 8, no. 1, pp. 1–12, 2019.
  * [19] M. Zaharia, M. Chowdhury, M. J. Franklin, S. Shenker, and I. Stoica, “Spark: Cluster computing with working sets.” _HotCloud_ , vol. 10, no. 10-10, p. 95, 2010.
  * [20] D. J. Reck, H. Haitao, S. Guidon, and K. W. Axhausen, “Explaining shared micromobility usage, competition and modechoice by modelling empirical data from Zurich, Switzerland,” _Transportation Research Part C: Emerging Technologies_ , vol. 124, p. 102947, 2021.
  * [21] P. A. Lopez, M. Behrisch, L. Bieker-Walz, J. Erdmann, Y.-P. Flötteröd, R. Hilbrich, L. Lücken, J. Rummel, P. Wagner, and E. Wießner, “Microscopic traffic simulation using sumo,” in _2018 21st International Conference on Intelligent Transportation Systems (ITSC)_. IEEE, 2018, pp. 2575–2582.
  * [22] K. W. Axhausen, A. Horni, and K. Nagel, _The multi-agent transport simulation MATSim_. Ubiquity Press, 2016\.
  * [23] L. Caggiani and M. Ottomanelli, “A modular soft computing based method for vehicles repositioning in bike-sharing systems,” _Procedia-Social and Behavioral Sciences_ , vol. 54, pp. 675–684, 2012.
  * [24] ——, “A dynamic simulation based model for optimal fleet repositioning in bike-sharing systems,” _Procedia-Social and Behavioral Sciences_ , vol. 87, pp. 203–210, 2013.
  * [25] R. M. Saltzman and R. M. Bradford, “Simulating a more efficient bike sharing system,” _Journal of Supply Chain and Operations Management_ , vol. 14, no. 2, p. 36, 2016.
  * [26] Y.-K. Lin and F. Liang, “Simulation for balancing bike-sharing systems,” _International Journal of Modeling and Optimization_ , vol. 7, no. 1, p. 24, 2017.
  * [27] W. Liang, J. Hao, and L. Zhang, “Travel Behavior Analysis for Free-Floating Bike Sharing Systems Based on Markov-Chain Models,” in _International Symposium on Positive Systems_ , 2018, pp. 127–145.
  * [28] C. Hebenstreit and M. Fellendorf, “A dynamic bike sharing module for agent-based transport simulation, within multimodal context,” _Procedia computer science_ , vol. 130, pp. 65–72, 2018.
  * [29] F. Soriguera, V. Casado, and E. Jiménez, “A simulation model for public bike-sharing systems,” _Transportation Research Procedia_ , vol. 33, pp. 139–146, 2018.
  * [30] H. Zhang, C. J. Sheppard, T. E. Lipman, T. Zeng, and S. J. Moura, “Charging infrastructure demands of shared-use autonomous electric vehicles in urban areas,” _Transportation Research Part D: Transport and Environment_ , vol. 78, p. 102210, 2020.
  * [31] C. Sheppard, R. Waraich, A. Campbell, A. Pozdnukov, and A. R. Gopal, “Modeling plug-in electric vehicle charging demand with BEAM: The framework for behavior energy autonomy mobility,” 2017.


# Revisiting Sample Size Determination in Natural Language Understanding

Ernie Chang†11footnotemark: 1 , Muhammad Hassan Rashid‡11footnotemark: 1 ,
Pin-Jie Lin‡11footnotemark: 1 ,
Changsheng Zhao†, Vera Demberg‡, Yangyang Shi† and Vikas Chandra†
${\dagger}$Reality Labs, Meta Inc.
$\ddagger$Saarland Informatics Campus, Saarland University, Germany
{erniecyc, cszhao, yyshi<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Knowing exactly how many data points need to be labeled to achieve a certain
model performance is a hugely beneficial step towards reducing the overall
budgets for annotation. It pertains to both active learning and traditional
data annotation, and is particularly beneficial for low resource scenarios.
Nevertheless, it remains a largely under-explored area of research in NLP. We
therefore explored various techniques for estimating the training sample size
necessary to achieve a targeted performance value. We derived a simple yet
effective approach to predict the maximum achievable model performance based
on small amount of training samples – which serves as an early indicator
during data annotation for data quality and sample size determination. We
performed ablation studies on four language understanding tasks, and showed
that the proposed approach allows us to forecast model performance within a
small margin of mean absolute error ($\backsim 0.9$%) with only 10% data111Our
code is available at: https://github.com/pjlintw/sample-size..

$\ast$ These authors contributed equally to this work.

##  1 Introduction

Labeled data play an important role in creating performant machine learning
models, which makes data annotation a fundamental process for any natural
language application pipeline Lewis and Catlett (1994); Chang et al. (2020).
Recent work has sought to reduce the annotation costs through the use of
active learning Ducoffe and Precioso (2018); Margatina et al. (2021) and data
sampling Sener and Savarese (2018); Coleman et al. (2019); Killamsetty et al.
(2021a, b); Chang et al. (2021). Indeed, these approaches are shown to be
effective in identifying or constructing data subsets needed to achieve a
competitive model performance. For instance, the active learning paradigm adds
new data iteratively to the existing set before model retraining Agarwal et
al. (2020); Margatina et al. (2021), improving upon the traditional human
annotation pipeline that obtains the entire labeled set all at once.

Nevertheless, the data labeling process typically annotates as much data as
the annotation budget permits, or by clearly defined stopping criteria to
terminate the labeling process. Unfortunately, this is usually challenging as
annotators do not have the knowledge of the effect of added labels to model
performance nor how much more data is needed to arrive at the desired model
generalizability Killamsetty et al. (2020). The stopping condition is in fact
tied to the quality of data samples w.r.t. model parameters Hu et al. (2021),
which influences the effective sample size222It is the size of datasets which
could have been achieved by an effective unweighted random sample Guo et al.
(2022)., and it is then beneficial to obtain an approximation of the expected
performance Vlachos (2008); Olsson and Tomanek (2009a); Zhu et al. (2010);
Ishibashi and Hino (2020). Therefore, knowing the approximate amount of
training data needed for this particular performance would serve as an useful
knowledge not only for deciding when to stop adding labeled data, but also as
an early indication for the data quality. For instance, by having early label
quality signals, we can decide between two different types of annotation, or
even between two pools of annotators with different expertise.

To this end, we explored the relationship between _data sample size_ and
_model performance_ in the context of language understanding via learning
curve modeling, which defines model performance as a function of dataset
sizes. By modeling this relationship in low resource settings, we obtain
useful early signals with approximated accuracies for any given the labeled
set, which can provide an idea for the sample size and data quality Olsson and
Tomanek (2009b); Figueroa et al. (2012). Previous studies have shown that
nonlinear weighted curve fitting methods such as inverse power laws or
exponential functions can provide decent approximations of the empirical
predictive performances Frey and Fisher (1999); Figueroa et al. (2012). We
thus put forward an ensemble of these functions which we showed to display a
consistently highly correlated behavior across four language understanding
benchmarks and with as little as 10% of the entire training set. This work
makes the following contributions:

  1. 1.

We revisit the task of sample size determination in four natural language
understanding benchmarks and empirically explore the correlation strengths of
several successful techniques.

  2. 2.

Based on our findings, we propose an Ensemble function and demonstrated across
several benchmarks and low resource settings that the ensemble function is
consistently providing a high correlation with the empirical learning curve
plots.

##  2 Background

Our method is a sample size determination technique that helps to design
annotation projects by determining the necessary sample size. Previous methods
have focused on identifying the sample size required to reach a specific
target performance, such as a high correlation coefficient Beal (1989);
Stalbovskaya et al. (2007); Beal (1989), which often involves predicting the
sample size necessary for a classifier to attain a specific accuracy level
Fukunaga and Hayes (1989). There are two main approaches for predicting the
sample size needed to achieve a particular classifier performance: (1) Dobbin
et al. (2008) present a model-based method for predicting the number of
samples required for classifying microarray data. (2) A more general approach
involves fitting a classifier’s learning curve to inverse power law models
Figueroa et al. (2012). Examples of this approach include algorithms proposed
by Mukherjee et al. (2003); Boonyanunta and Zeephongsekul (2004); Last (2007).

##  3 The Approach

#### Learning Curve Modeling.

A learning curve is a graphical representation of how a classifier’s
performance changes as the size of the training set increases. The curve
typically has three sections: an initial section where performance improves
rapidly with increasing training set size, a middle section where the rate of
improvement slows down, and a final section where the classifier reaches its
maximum performance and further increases in training set size do not lead to
significant improvements. This relationship can be quantified using a set of
data points, each of which represents the expected performance of the
classifier $E_{acc}$ on a particular training set size $D_{k}$. These data
points can be plotted to create the learning curve, which can help to
understand the behavior of the classifier and inform decision-making about how
much training data is needed to achieve a desired performance level.

#### Task Description.

Given a downstream classification task with $N_{total}$ data points, a
learning curve model $F$ predicts the expected performance $E_{acc}$ when a
classifier trained on the an observed range of training set size ($D_{k}$;
$k>=N$). The empirical learning curve is assessed by the parametric models for
the learning algorithm performance extrapolation. In our settings, we set
$k<<N_{total}$ to simulate practical settings, where few data points
consisting of $(E_{acc},D_{K})$ are to be obtained.

#### Types of Extrapolations.

Here, we study different forms of learning curve models with few learnable
parameters that have been proven as simple yet effective. The simplest type of
learning curve model _exponential function_ (Exp) only introduces two
parameters $a$ and $b$ to fit the exponent behavior of learning curve Frey and
Fisher (1999). The second form, _Inverse Power Law function_ (Inverse), fits
the inverse power law Figueroa et al. (2012) and has three parameters. The
third form uses a function from the power law family – Power4 function (Pow4)
Kolachina et al. (2012) with four parameters. Lastly, we propose to combine
all functions into one (Ensemble) so that it has all their characteristics in
order to make it more robust across benchmarks. Table 1 shows the formulae of
our investigated extrapolating functions.

Extrapolating Functions | Formula
---|---
exp (A) | $a\cdot N^{b}$
inverse (B) | $(1-a)-b\cdot N^{c}$
pow4 (C) | $a-(b\cdot N+c)^{-d}$
Ensemble (A+B+C) | $-$
|

Table 1: Overview of extrapolating functions

##  4 Experimental Settings

We study four NLU tasks: (1) IMDb Maas et al. (2011) is a binary
classification dataset ($25$K/–/$25$K)333Expressed in the order
(train/dev/test). where model predicts the sentiment (positive/negative) for
movie reviews from IMDB; (2) SST2 Socher et al. (2013) is also a sentiment
classification datatset ($67$K/$0.8$K/$1.8$K) containing reviews of different
movies and since the model predicts if the review is positive or negative, it
also falls in the category of binary classification; (3) AG NEWS is a multi-
class classification dataset ($120$K/–/$7.6$K) containing texts from different
news where the model predicts whether the news text is about sports,
science/technology, world or business from the four different classes. We also
consider one other multi-class classification task, (4) DBpedia dataset
($560$K/–/$70$K) , since it could help us in testing the robustness of the
methods used in our experiments.

#### Configs.

To investigate how changes in data size affect the predictiveness of the
learning curves, under the assumption that the model structure and settings
remain unchanged, we perform all experiments using a transformer model Vaswani
et al. (2017) and average the results over 3 initialization runs. The
embedding and hidden layer dimensions are 1000 and 1820; and we use a 6-layer
encoder with 4 multi-heads, and the dropout is 0.2. To find the parameters of
learning curve models, we consider unweighted and for the gradient descent and
non-linear least squares optimizers. The Adam algorithm Kingma and Ba (2014)
was used as the optimizer with learning rate of 1e-5 and ReLU was used as the
activation function. The cross-entropy objective was used for all
classification benchmarks, and we select the models using loss values.
Finally, we chose a batch size of 8 with 200 number of epochs.

#### Evaluation.

We use the aforementioned functions: Exp, Inverse, Pow4 and Ensemble for
fitting the empirical learning curve. For each dataset, we select training set
sizes ranging from $1$% to $10$% data sizes at an interval of $1$%. The
learning curve testsets were created with the data splits in the range
$[55,100]$ at $5$% interval by training the classifier, and obtaining the
testset444Here, we make the distinction between testset for learning curve and
the original testset split. performance for each corresponding data split.
Therefore, we collect the accuracies against different sample sizes and report
the mean absolute error (MAE) as the evaluation metric for learning curve
modeling.

##  5 Results and Analysis

We present results of ensemble method for learning curve modeling on the NLU
benchmarks.

###  5.1 Main Results

Figure 1 demonstrates that by using only 10% of the data for learning curve
modeling, Ensemble is able to effectively predict model performance within a
0.9% margin of the actual model performance. Moreover, we observe the same
trend across all four benchmarks consisting of different training set sizes
(i.e. ranging from 25K to 250K) and varying number of classification classes
(i.e. ranging from 2 to 14), see the appendix A for remaining figures. Our
result shows that the proposed approach is not confined by the classification
types and sample sizes.

Table 2 shows the saturated points of the learning curve when the performance
improvement is less than a threshold $\alpha=0.2$ – we found that the
predicted performance with only $19$% data is within $2.44$ accuracy points
from the trained model performance for IMDb. Another key observation is that
the size (%) needed to predict a low L1 distance increases as the number of
classification classes goes up, which indicates that task difficulty does
influence the ease of extrapolation. An example is that AG News requires up to
$51$% to predict a low L1 distance. Next, we perform further ablation studies
to investigate the effect of sample size, types of non-linear functions used,
or the effect of data weighting.

Benchmark | Cls (#N) | Size (%) | Size (#N) | L1$\downarrow$ | 100%
---|---|---|---|---|---
c
IMDb | $2$ |  $36$% | $6,300$ | $2.44$ |  $17$K
SST2 | $2$ |  $19$% | $8,958$ | $5.57$ |  $47$K
AG News | $4$ |  $51$% | $42,840$ | $2.6$ |  $84$K
DBpedia | $14$ |  $51$% | $199,920$ | $2.39$ |  $392$K
|  |  |  |  |

Table 2: Cls (#N) stands for the number of classes, Size (%) for the
percentages of the data size for the learning curve modeling. Size (#N) is the
number of the corresponding data size, L1 is the L1 distance between the
accuracy of models using all the training data and the estimated accuracy
based on the saturated point. 100% specifies all training samples for learning
curve.

Figure 1: Learning curves on $10\%$ sample size of IMDB and SST2 datasets. We
plot learning curves using the exponential (Exp), inverse power law (Inv),
power4 (Pow4) function, and the combination of the aforementioned forms
(Ensemble). The learning curves only fit on $10\%$ training sample (blue) and
generalize on the unseen data sizes. We evaluate the learning curves on the
testing sample (yellow). Data sizes from $10\%$ to $50\%$ (teal) are neither
used in training nor testing.

###  5.2 Ablation Study

#### Effect of sample size.

In Figure 1, we study the correlation between sample sizes and the absolute
mean error between the learning curve model and empirical model performance
trend. Surprisingly, we discovered by having more samples does not necessarily
help with modeling a better learning curve555We showed this result in the
Appendix A.5., and that with only $10$% data to build the $(D_{k},E_{acc})$
data points is sufficient to obtain rather small errors across all four
benchmarks.

#### Types of learning curve functions.

We are also interested in seeing how each of the non-linear learning curve
function fare against each other in simpler settings. To this end, we used up
to 10% data to model the learning curves and obtained their respective mean
absolute error values. In Figure 1, we present this comparison where we showed
that on IMDb and SST2, the Ensemble function consistently fit best against the
empirical data. We observed a similar trend across other benchmark DBpedia
with the exception of AG NEWS. We placed the plot for AG NEWS in appendix A.3.

#### Influence of data weighting.

Previous work Paul et al. (2021); Guo et al. (2022) has found that not all
data points are equally important in terms of curve fitting. In fact, data
points at a later phase corresponding to more samples are to be given more
weight compared to earlier points. We thus investigate this phenomenon in the
context of our benchmark, and we observed this to be true anecdotally. The
detailed result can be found in Appendix A.2. The reason for this is that the
more data samples there are, the more closely they resemble the entire
training set, and this makes their signals a better estimation of a point on
the actual learning curve. Another perspective is that the more data samples
are used, the less the effect of random sampling on the performance, which
affects model performance in extremely low resource scenarios.

Function Type |  c
---|---
Unweighted | Weighted
Exp | $0.0417$ | 0.0244
Inv | $0.00777$ | 0.00442
Pow4 | $0.00795$ | $0.00795$
|  |

Table 3: Better curve fitting when weighting data points at latter phase. We
examine the effectiveness of weighting data size on the exponential (Exp),
inverse power law (Inv), power4 (Pow4) function using non-linear least squares
method. The learning curves fit on 5%, 10%, 25% and 50% data sizes of IMDB and
is evaluated on testing sample with mean absolute error (MAE).

##  6 Conclusions and Future Works

In this work, we investigated techniques for estimating the amount of training
data needed to achieve a target performance in four natural language
understanding benchmarks. We demonstrated that our approach allows for
accurate prediction of model performance using only a small portion of the
data, which can be useful in scenarios with limited resources. Nevertheless,
we also recognize the limitation in our current study. For instance, we did
not explore sampling techniques other than random sampling; while recent works
Yuan et al. (2020); Paul et al. (2021); Guo et al. (2022) have shown promising
directions in data sampling that outperforms random sampling. Another
interesting direction is to explore the model architecture’s influence on
generalizability, and thus the learning curve, which we left for future works.

## Limitations

While the effectiveness of the expressive learning curve in settings with
limited data has been demonstrated, it is uncertain if this success can be
replicated in more complex natural language understanding tasks, such as
question answering or tasks that involve a large amount of data. Furthermore,
it is assumed that all data samples have the same impact on the model’s
performance. However, the actual performance of the model may vary based on
the method used to select the data or the specific set of tasks being
performed, e.g., coreset selection. Similarly, the quality of the labels used
for the data can also play a significant role in predicting the performance of
the model. Overall, we plan to further investigate these questions and explore
them in future studies.

## Ethics Statement

We address the efficiency of data annotation by investigating learning curves
to estimate the necessary training sample size to reach a desired model
performance. However, it is imperative to take into consideration the
potential biases that may exist in the model predictions when utilizing a
reduced amount of labeled data in the system construction process.
Furthermore, when addressing complex tasks such as machine translation and
text summarization, it is essential to guarantee the factuality of output
generated by the system trained with the suggested data sample size.

## References

  * Agarwal et al. (2020) Sharat Agarwal, Himanshu Arora, Saket Anand, and Chetan Arora. 2020.  Contextual diversity for active learning.  In _ECCV_ , pages 137–153. Springer.
  * Beal (1989) S L Beal. 1989.  Sample size determination for confidence intervals on the population mean and on the difference between two population means.  _Biometrics_ , 45(3):969–977.
  * Boonyanunta and Zeephongsekul (2004) Natthaphan Boonyanunta and Panlop Zeephongsekul. 2004.  Predicting the relationship between the size of training sample and the predictive power of classifiers.  _Knowledge-Based Intelligent Information and Engineering Systems_ , 3215:529–535.
  * Chang et al. (2020) Ernie Chang, Jeriah Caplinger, Alex Marin, Xiaoyu Shen, and Vera Demberg. 2020.  DART: A lightweight quality-suggestive data-to-text annotation tool.  In _Proceedings of the 28th International Conference on Computational Linguistics: System Demonstrations_ , pages 12–17, Barcelona, Spain (Online). International Committee on Computational Linguistics (ICCL).
  * Chang et al. (2021) Ernie Chang, Xiaoyu Shen, Hui-Syuan Yeh, and Vera Demberg. 2021.  On training instance selection for few-shot neural text generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 8–13, Online. Association for Computational Linguistics.
  * Coleman et al. (2019) Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. 2019.  Selection via proxy: Efficient data selection for deep learning.  _arXiv preprint arXiv:1906.11829_.
  * Dobbin et al. (2008) Kevin K Dobbin, Yingdong Zhao, and Richard M Simon. 2008.  How large a training set is needed to develop a classifier for microarray data?  _Clin. Cancer Res._ , 14(1):108–114.
  * Ducoffe and Precioso (2018) Melanie Ducoffe and Frederic Precioso. 2018.  Adversarial active learning for deep networks: a margin based approach.  _arXiv preprint arXiv:1802.09841_.
  * Figueroa et al. (2012) Rosa L Figueroa, Qing Zeng-Treitler, Sasikiran Kandula, and Long H Ngo. 2012.  Predicting sample size required for classification performance.  _BMC Med Inform Decis Mak_ , 12.
  * Frey and Fisher (1999) Lewis J. Frey and Douglas H. Fisher. 1999.  Modeling decision tree performance with the power law.  In _Proceedings of the Seventh International Workshop on Artificial Intelligence and Statistics_ , volume R2 of _Proceedings of Machine Learning Research_. PMLR.  Reissued by PMLR on 20 August 2020.
  * Fukunaga and Hayes (1989) K Fukunaga and R R Hayes. 1989.  Effects of sample size in classifier design.  _IEEE Trans. Pattern Anal. Mach. Intell._ , 11(8):873–885.
  * Guo et al. (2022) Chengcheng Guo, Bo Zhao, and Yanbing Bai. 2022.  Deepcore: A comprehensive library for coreset selection in deep learning.
  * Hu et al. (2021) Xia Hu, Lingyang Chu, Jian Pei, Weiqing Liu, and Jiang Bian. 2021.  Model complexity of deep learning: A survey.  _Knowledge and Information Systems_ , 63(10):2585–2619.
  * Ishibashi and Hino (2020) Hideaki Ishibashi and Hideitsu Hino. 2020.  Stopping criterion for active learning based on deterministic generalization bounds.  In _Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics_ , volume 108 of _Proceedings of Machine Learning Research_ , pages 386–397. PMLR.
  * Killamsetty et al. (2021a) Krishnateja Killamsetty, S Durga, Ganesh Ramakrishnan, Abir De, and Rishabh Iyer. 2021a.  Grad-match: Gradient matching based data subset selection for efficient deep model training.  In _ICML_ , pages 5464–5474.
  * Killamsetty et al. (2020) KrishnaTeja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, and Rishabh K. Iyer. 2020.  GLISTER: generalization based data subset selection for efficient and robust learning.  _CoRR_ , abs/2012.10630.
  * Killamsetty et al. (2021b) Krishnateja Killamsetty, Xujiang Zhao, Feng Chen, and Rishabh Iyer. 2021b.  Retrieve: Coreset selection for efficient and robust semi-supervised learning.  _arXiv preprint arXiv:2106.07760_.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Kolachina et al. (2012) Prasanth Kolachina, Nicola Cancedda, Marc Dymetman, and Sriram Venkatapathy. 2012\.  Prediction of learning curves in machine translation.  In _Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 22–30, Jeju Island, Korea. Association for Computational Linguistics.
  * Last (2007) Mark Last. 2007.  Predicting and optimizing classifier utility with the power law.  _Seventh IEEE International Conference on Data Mining Workshops_ , pages 219–224.
  * Lewis and Catlett (1994) David D Lewis and Jason Catlett. 1994.  Heterogeneous uncertainty sampling for supervised learning.  In _Machine learning proceedings 1994_ , pages 148–156. Elsevier.
  * Maas et al. (2011) Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. 2011.  Learning word vectors for sentiment analysis.  In _Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies_ , pages 142–150.
  * Margatina et al. (2021) Katerina Margatina, Giorgos Vernikos, Loïc Barrault, and Nikolaos Aletras. 2021\.  Active learning by acquiring contrastive examples.  _arXiv preprint arXiv:2109.03764_.
  * Mukherjee et al. (2003) Sayan Mukherjee, Pablo Tamayo, Simon Rogers, Ryan Rifkin, Anna Engle, Colin Campbell, Todd R Golub, and Jill P Mesirov. 2003.  Estimating dataset size requirements for classifying dna microarray data.  _Comput Biol_ , 10:119–142.
  * Olsson and Tomanek (2009a) Fredrik Olsson and Katrin Tomanek. 2009a.  An intrinsic stopping criterion for committee-based active learning.  In _Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL-2009)_ , pages 138–146, Boulder, Colorado. Association for Computational Linguistics.
  * Olsson and Tomanek (2009b) Fredrik Olsson and Katrin Tomanek. 2009b.  An intrinsic stopping criterion for committee-based active learning.  _Proceedings of the Thirteenth Conference on Computational Natural Language Learning_ , pages 138–146.
  * Paul et al. (2021) Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. 2021.  Deep learning on a data diet: Finding important examples early in training.  _Advances in Neural Information Processing Systems_ , 34:20596–20607.
  * Sener and Savarese (2018) Ozan Sener and Silvio Savarese. 2018.  Active learning for convolutional neural networks: A core-set approach.  In _ICLR_.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Y. Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1631–1642.
  * Stalbovskaya et al. (2007) Viktoriya Stalbovskaya, Brahim Hamadicharef, and Emmanuel C Ifeachor. 2007.  Sample size determination using ROC analysis.  In _3rd International Conference on Computational Intelligence in Medicine and Healthcare (CIMED2007)_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _CoRR_ , abs/1706.03762.
  * Vlachos (2008) Andreas Vlachos. 2008.  A stopping criterion for active learning.  _Computer Speech and Language_ , 22(3):295–312.
  * Yuan et al. (2020) Michelle Yuan, Hsuan-Tien Lin, and Jordan Boyd-Graber. 2020.  Cold-start active learning through self-supervised language modeling.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7935–7948, Online. Association for Computational Linguistics.
  * Zhu et al. (2010) Jingbo Zhu, Huizhen Wang, Eduard Hovy, and Matthew Ma. 2010.  Confidence-based stopping criteria for active learning for data annotation.  _ACM Trans. Speech Lang. Process._ , 6(3).

##  Appendix A Detailed Results

###  A.1 Predicting the Required Data Size

Table 4 presents the results of required data size prediction using threshold
$\alpha=0.1$ and $\alpha=0.3$.

Benchmark | Cls (#) | Size (%) | Size (#N) | L1$\downarrow$ | 100%
---|---|---|---|---|---
c
IMDb | $2$ |  $19$% | $16,800$ | $6.56$ |  $17$K
SST2 | $2$ |  $8$% | $25,458$ | $8.27$ |  $47$K
AG News | $4$ |  $28$% | $82,320,$ | $2.96$ |  $84$K
DBpedia | $14$ |  $27$% | $384,160$ | $3.44$ |  $392$K
c
IMDb | $2$ |  $96$% | $3,325$ | $5.84$ |  $17$K
SST2 | $2$ |  $54$% | $3,772$ | $0.704$ |  $47$K
AG News | $4$ |  $98$% | $23,521$ | $9.9$ |  $84$K
DBpedia | $14$ |  $98$% | $105,840$ | $9.68$ |  $392$K
|  |  |  |  |

Table 4: We show the results with the thresholds $\alpha=0.1$ and
$\alpha=0.3$. Size (%) stands for the percentages of the data size for the
learning curve modeling, Size (#N) is the number of the corresponding data
size, L1 is the L1 distance between the accuracy of models using all the
training data and the estimated accuracy based on the saturated point. 100%
specifies all training samples.

###  A.2 Data Weighting

We apply data weighting on three extrapolating functions using gradient decent
methods in 5.

Extrapolating |  c
---|---
Unweighted | Weighted
Exp | $0.0417$ | 0.0342
Inv | $0.0706$ | 0.0519
Pow4 | $0.0979$ | 0.0652
|  |

Table 5: Better curve fitting when weighting data points at latter phase. We
examine the effectiveness of weighting data size on the exponential (Exp),
inverse power law (Inv), power4 (Pow4) function using gradient decent method.
The learning curves fit on 5%, 10%, 25% and 50% data sizes of IMDB and is
evaluated on testing sample with mean absolute error (MAE).

Figure 2: Learning curves on $10\%$ data size using AG NEWS.

###  A.3 Learning curve on 10% data sizes of AG NEWS

Figure 2 shows the learning curves fitting on $10$% data sizes of AG NEWS
dataset.

Figure 3: Learning curves on $10\%$ data size using DBpedia.

###  A.4 Learning curve on 10% data sizes of DBpedia

Figure 3 shows the learning curves fitting on $10$% data sizes of DBpedia
dataset.

###  A.5 Effect of Sample Sizes for Learning Curve Fitting

We examined the relationship between sample sizes and the difference in mean
absolute error (MAE) between the predicted and actual performance trends
across four benchmarks. Table 6 showed MAEs when Ensemble fitting on $50\%$
and $10\%$ of data respectively. We observed that having more samples does not
necessarily lead to a better model and that using only 10% resulted in smaller
MAEs on all four benchmarks. Therefore, we select 10% of data points for
learning curve modeling.

Benchmark |  c
---|---
50% | 10%
IMDb | $0.0458$ | 0.00961
SST2 | $0.0299$ | 0.0132
AG News | $0.0704$ | 0.0209
DBpedia | $0.0734$ | 0.0158
|  |

Table 6: Learning Curve Fitting on $50\%$ and $10\%$ data size respectively.