# A System for Accessible Artificial Intelligence

Randal S. Olson  Institute for Biomedical Informatics, University of
Pennsylvania, Philadelphia, PA 19104  Moshe Sipper  Institute for Biomedical
Informatics, University of Pennsylvania, Philadelphia, PA 19104  Department of
Computer Science, Ben-Gurion University, Beer-Sheva 8410501, Israel  William
La Cava  Institute for Biomedical Informatics, University of Pennsylvania,
Philadelphia, PA 19104  Sharon Tartarone  Institute for Biomedical
Informatics, University of Pennsylvania, Philadelphia, PA 19104  Steven Vitale
Institute for Biomedical Informatics, University of Pennsylvania,
Philadelphia, PA 19104  Weixuan Fu  Institute for Biomedical Informatics,
University of Pennsylvania, Philadelphia, PA 19104  Patryk Orzechowski
Institute for Biomedical Informatics, University of Pennsylvania,
Philadelphia, PA 19104  Department of Automatics and Biomedical Engineering,
AGH University of Science and Technology, al. Mickiewicza 30, 30-059 Krakow,
Poland  Ryan J. Urbanowicz  Institute for Biomedical Informatics, University
of Pennsylvania, Philadelphia, PA 19104  John H. Holmes  Institute for
Biomedical Informatics, University of Pennsylvania, Philadelphia, PA 19104
Jason H. Moore  Institute for Biomedical Informatics, University of
Pennsylvania, Philadelphia, PA 19104

###### Abstract

While artificial intelligence (AI) has become widespread, many commercial AI
systems are not yet accessible to individual researchers nor the general
public due to the deep knowledge of the systems required to use them. We
believe that AI has matured to the point where it should be an accessible
technology for everyone. We present an ongoing project whose ultimate goal is
to deliver an open source, user-friendly AI system that is specialized for
machine learning analysis of complex data in the biomedical and health care
domains. We discuss how genetic programming can aid in this endeavor, and
highlight specific examples where genetic programming has automated machine
learning analyses in previous projects.

##  1 Introduction

A central goal of artificial intelligence (AI) is to use computational
hardware and software to solve complex problems in a human-competitive manner
[9]. The practicality of this goal is that AI can be tasked with solving
problems or performing functions that humans cannot perform or simply don’t
have time for. Most AI methodologies can be grouped into top-down approaches,
wherein cognition is viewed as a high-level phenomenon that is independent of
the low-level details, or bottom-up approaches, which define basic
computational building blocks such as artificial neurons that collectively
give rise to “emergent” [29] intelligent behavior. The top-down approach has
been difficult to realize given the inherent complexity of human cognition.
However, the bottom-up has had some success owing to the availability of
sophisticated algorithms such as genetic programming (GP) [10] and deep neural
networks [6]. This is particularly true today with abundant and inexpensive
high-performance computing, leading to many human-competitive success stories
[9].

Medical applications of AI have had a long history with both successes and
failures. One of the early successes was the Mycin system, which was designed
to predict the antibiotic that a patient with an infection should receive in
the intensive care unit [2]. Mycin combined a knowledge base along with a set
of rules implemented as part of an expert system. The system was demonstrated
to be human-competitive, but was never put into clinical practice because of
legal concerns and the time it took clinicians to enter the patient data
required for Mycin to make the predictions. The field of AI has matured since
Mycin was developed and, importantly, computing power has grown tremendously
in parallel. Examples of modern AI successes include IBM’s Watson, which beat
the world champion of the game show Jeopardy [5]. The Watson AI system that
won Jeopardy combined knowledge representation, information retrieval, natural
language processing, and machine learning along with high-performance
computing to access and exploit a knowledge base that included the Wikipedia
text corpus. This was a milestone in AI because it showed that a computational
system could compete with humans on difficult language processing tasks.
Watson is now being marketed in the health care domain although the jury is
still out on its effectiveness.

Commercial AI systems such as Watson show potential but are not yet accessible
to individual researchers nor the general public due the cost and the
complexity of working with a team from IBM. It is our working hypothesis that,

> AI has matured to the point where it should be an accessible technology for
> everyone.

Democratization of AI will be important if we seek to integrate this exciting
new technology into multiple different domains, as demonstrated by recent
efforts such as Orange [4]. We describe here the early development stages of
an open source and user-friendly AI system—PennAI (http://pennai.org)—for
machine learning analysis of complex data in the biomedical and health care
domains. We focus our initial efforts on the classification of biomedical
endpoints such as disease susceptibility. We describe in turn below each of
the components of our AI system and then end with an example and a discussion
of how we envision this system being used to solve complex biomedical
problems. Further, we discuss how GP can aid in enhancing PennAI, and
highlight specific examples where GP has automated machine learning analyses
in previous work.

The components of PennAI include a human engine (i.e., the user); a user-
friendly interface for interacting with the AI; a machine learning engine for
data mining; a controller engine for launching jobs and keeping track of
analytical results; a graph database for storing data and results (i.e., the
memory); an AI engine for monitoring results and automatically launching or
recommending new analyses; and a visualization engine to displaying results
and analytical knowledge (Figure 1). This AI system provides a comprehensive
set of integrated components for automated machine learning (AutoML), thus
providing a data science assistant for generating useful results from large
and complex data problems. PennAI is housed in the “Idea Factory,” a facility
designed to facilitate collaboration and promote new methods of communicating
and presenting scientific innovation. The Idea Factory makes sophisticated
data visualization and artificial intelligence analytics easy for users across
the entire Penn community (Figure 2).

Figure 1: The components of PennAI, a user-friendly AI system developed at the
University of Pennsylvania. Figure 2: The “Idea Factory,” home of PennAI.

##  2 The Human Engine

The most important component of the proposed AI system is the user. Contrary
to some claims that AI will replace human users, we see the human as an
integral part of the discovery process and a partner with the AI. One way to
view this partnership is with the human as the driver of the discovery process
and the AI as the data science assistant. Thus, the AI provides an additional
set of hands in a modern data science discovery environment that might include
human teammates with expertise in computer science, statistics, and applied
mathematics. We have previously suggested this idea of human-computer
interaction that places the human user at the epicenter [22]. This idea has
also previously been explored from the point of view of the user or domain
expert [16].

Langley [16] provides five important tips that are relevant to thinking about
the relationship humans have with AI for data mining using machine learning.
First, traditional machine learning notations are not easily communicated to
scientists. This consideration is important because a machine learning model
may not be interpretable by a user. Second, scientists often have initial
models that should influence the discovery process. Domain-specific knowledge
can be critical to the discovery process. Third, scientific datasets are often
rare and difficult to obtain. It often takes years to collect and process the
data before it can be analyzed. As such, it is important that the analysis is
carefully planned and executed, and that any general feedback about the
performance the learning process is not lost between studies. Fourth,
scientists want models that move beyond description and provide explanations
of the data. Explanation and interpretation are paramount to the user.
Finally, scientists want computational assistance rather than a complete
replacement of themselves. Langley [16] further suggests that users want
interactive discovery environments that help them understand their data while
at the same time giving them control over the modeling process. Collectively,
these five lessons suggest that synergy between the user and the AI is
critical. With this in mind, our proposed AI system includes a graphical user
interface (GUI) that allows the user to easily launch analyses, view the
results, and give the AI feedback about what results are useful or
interesting.

##  3 The Human-Computer Interaction Engine

As described above, a key component of PennAI is human-computer interaction.
The first important feature is to make it easy for the user to directly launch
machine learning analyses by choosing a method and its parameter settings from
an intuitive push-button menu implemented through the web using JavaScript.
The user can launch single analyses or, in an advanced mode, launch a grid
search across multiple methods and parameter settings. The methods and the
controller that keep track of these analyses is described below. Figures 3 and
4 show prototypes of our GUI for uploading and viewing datasets for analysis
and launching machine learning analyses on those datasets, respectively. Our
JavaScript implementation is compatible with mobile devices, which allows the
user to interact with the AI system from any Internet-connected device.

The second key feature of PennAI is the ability to toggle the AI on and off
for automated analysis, shown in Figure 3. An AI toggle allows the user to
turn the AI on and set parameters controlling the maximum number of runs the
AI can launch, as well as the frequency of updates the user would like to
receive by email or text message. The GUI also provides a simple thumbs
up/down selection for each result received by PennAI, which provides feedback
to PennAI that is incorporated into its expert knowledge system.

Figure 3: Prototype of the graphical user interface for managing and viewing
datasets.

##  4 The Machine Learning Engine

Our first application of PennAI is for data mining using machine learning in
the biomedical domain. Here, we make use of an extensive open source machine
learning library in Python called scikit-learn [28]. Scikit-learn provides
peer-reviewed implementations of several common supervised and unsupervised
machine learning algorithms, data preprocessing methods, feature engineering
and selection methods, hyperparameter optimization procedures, and more. To
most users, scikit-learn is considered to be the standard machine learning
library in Python.

Of course, there are dozens of machine learning algorithms, preprocessors,
etc. to choose from in scikit-learn, which can be overwhelming to a novice
user. To simplify the algorithm selection process for PennAI users, we
currently limit PennAI to six machine learning algorithms that we believe will
handle most supervised classification use cases, shown in Table 1. We also
limit the parameter choices for each algorithm to a handful of the most
important parameters and parameter options, which makes it easier for users to
choose a parameter configuration at the expense of algorithm customizability.
An example of the interface to the Machine Learning Engine can be found in
Figure 4, where only a handful of the most important parameters and parameter
options are available for the k-Nearest Neighbors classification algorithm.

Classification | Regression
---|---
Logistic Regression | ElasticNet
Decision Tree | Decision Tree
k-Nearest Neighbors | k-Nearest Neighbors
Support Vector Machine | Support Vector Machine
Random Forest | Random Forest
Gradient Boosting | Gradient Boosting
Table 1: Machine learning algorithms available in PennAI.

In an upcoming PennAI implementation, we will provide simplified descriptions
of the machine learning algorithms and parameters so users can make use of the
algorithms without fully understanding their implementation. For example, when
using a random forest it is not necessary for the user to understand what
tuning the n_estimators parameter does to the model. Instead, it is more
important for the user to understand that adding more decision trees to the
random forest (i.e., increasing n_estimators) improves model performance but
increases training time, whereas removing decision trees from the random
forest decreases model performance but decreases training time [7].

Once the Machine Learning Engine finishes training and evaluating a machine
learning model, it stores the machine learning model, the model predictions,
and an analysis of the model in the Graph Database Engine, which are used in
the Visualization Engine (both described below).

Figure 4: Prototype of the Machine Learning Engine graphical user interface.

##  5 The Controller Engine

The Controller Engine acts as the interface between the high-performance
computing system and the user or AI. This component is hidden from the user
but facilitates the automatic launching of jobs on a multi-CPU machine,
computing cluster, or cloud computing system. The controller must not only
coordinate the launching of jobs but also keep track of when they finish and
deposit the results in the Graph Database Engine (described below) that serves
as the memory of the system.

For the Controller Engine, we selected an open source package called the
Future Gadget Lab (FGLab), which is available on GitHub111FGLab:
https://github.com/Kaixhin/FGLab. FGLab functions as a server with individual
runs launched as clients, called FGMachines. FGLab uses node.js to coordinate
distributed jobs and uses MongoDB [3] as the backend database in the Graph
Database Engine.

##  6 The Graph Database Engine

Another key component of PennAI is a memory system that keeps track of every
analysis that is run on each data set. We keep track of the details of the
machine learning method, the parameter settings, the data set analyzed, and
results such as the model, model error, and area under the receiver operating
characteristic curve (AUC). These are all stored in a JSON file that is
deposited in a MongoDB NoSQL database. The advantage of using a NoSQL database
is that new data elements can be added without creating tables and without
strict format specifications. This flexibility is important for the rapidly
changing landscape of machine learning. MongoDB can also function as a graph
database that allows the documents to be linked in a network according to
shared index terms related to the analysis and data. This feature facilitates
more complex semantic queries of the database, such as “Return the machine
learning algorithm configurations that achieved the highest accuracy on any
study involving prostate cancer.”

###  6.1 Knowledge Base

The Graph Database Engine serves as the memory of PennAI and provides the raw
materials for the AI to learn which methods and parameter settings are working
better than others for particular kinds of problems. The initial knowledge
base consists of results from a previously published benchmark of scikit-learn
algorithms [24], in which 14 machine learning algorithms were run with full
hyperparameter optimization on a suite of 165 supervised classification
problems. The results are combined with meta-information about the datasets
(e.g., number of features, number of instances, correlations between features,
etc.) in order to allow the creation of a mapping from ‘problem instance
space’, i.e. dataset meta-features and model performance, to ‘learning space’,
i.e. machine learning algorithms and their parameters. This data can then be
modelled to extract rules that represent the knowledge used by the Artificial
Intelligence Engine to make informed analyses. The knowledge base will be
updated with all future analyses.

##  7 The Artificial Intelligence Engine

Each component described above provides the raw materials for the Artificial
Intelligence Engine which then 1) searches the graph database for results
related to one or more data sets, 2) performs statistical analysis comparing
algorithms and their parameters, 3) combines facts and rules in an expert
system to make new analysis recommendations, 4) communicates findings to the
user, and 5) automatically launches new analyses using suggestions from the
expert system. The first function uses the search capabilities of the MongoDB
graph database to identify relevant machine learning results in the form of
JSON files. All returned JSON files can be parsed to extract the machine
learning algorithm, parameters, and information about the model performance.
These results are collated in a tab-delimited file and a statistical analysis
performed to determine the best algorithm configurations for certain problem
types, similar to meta-learning techniques [8].

New statistical results are used to populate the knowledge base of an expert
system that has a set of decision rules provided by developers and advanced
machine learning practitioners. This expert system is then used to make
suggestions for additional analyses, for example by recommending better
parameter settings or even entirely different machine learning algorithms that
might be better-suited for the user’s dataset. The user can access these
suggestions manually or PennAI can use the suggestions to automatically launch
new jobs, thus continually growing the PennAI knowledge base. Essentially, the
Artificial Intelligence Engine becomes a research assistant who tinkers with
new ways of modeling the dataset and reports back to the user with their best
findings.

##  8 The Visualization Engine

Visualization will be critical for fostering the human-AI collaboration
described above. The user will need to be able to see individual machine
learning models and results as well as higher-level results from statistical
analyses across machine learning runs. We extract visual results such as the
receiver operating characteristic (ROC) curves and models to store in the
graph database, as shown in Figure 5. PennAI will also generate heatmaps and
other visualizations that summarize results across different machine learning
methods and datasets. These higher-level visualizations will aid the user with
making decisions about new manual analyses to launch and will help them assess
how well the PennAI assistant is doing. These images will be linked to the
datasets and results in the Graph Database Engine, and will thus be easily
searchable.

Figure 5: Prototype of the Visualization Engine graphical user interface.

##  9 Discussion and Future Work

Thus far, we have described PennAI as a system that provides a simple
interface for users to upload their datasets, launch machine learning
analyses, view the results of the analyses in an intuitive manner, and use
those results to refine their machine learning analyses. We also described how
PennAI will use a combination of expert knowledge from advanced machine
learning practitioners and prior statistical knowledge of machine learning
algorithm performance on datasets to recommend new analyses to the user, as
well as launch its own analyses to later report to the user. In essence, the
primary goal of PennAI is to provide an AI research assistant for its users.
However, considering the name of this workshop and book—Genetic Programming
Theory and Practice—one may be left wondering how GP can be incorporated into
PennAI. In the following paragraphs, we will describe our plans for
integrating GP into PennAI.

The first point of entry is to include GP as a machine learning option since a
number of successful biomedical applications have been reported (e.g., [34,
19, 17, 20, 18, 21]). A GP system for classification based on multidimensional
clustering [31] was recently demonstrated on biomedical classification
problems [15] as a competitive alternative to traditional machine learning
approaches. Recently GP has been proposed as a general feature engineering
wrapper (FEW)222http://lacava.github.io/few in order to harness its feature
learning capability to improve scikit-learn estimators, both for regression
[13] and classification [14]. FEW allows GP to provide readable feature
transformations to users while still utilizing existing modeling techniques
for making predictions. As mentioned in Section 2, interpretation and
explanatory power are important aspects of using AI for data mining, and
therefore GP methods that produce concise models, e.g. by local search [11] or
Pareto optimization [12], are important options to include. Further down the
road, it could be possible for PennAI to allow advanced users to incorporate
custom machine learning algorithms into PennAI by providing a scikit-learn
formatted interface to their project (e.g.
ellyn333http://epistasislab.github.io/ellyn). PennAI could then provide a
“bring your own learner” type of service [1] to allow researchers to tackle
complex data mining tasks with customized learning approaches, and incorporate
the results into its knowledge base for improving future data science
projects.

Beyond using GP to perform the machine learning itself, recent work has shown
that GP can also be harnessed to optimize a sequence of existing data analysis
and machine learning operations on a dataset to maximize the predictive
performance of the final machine learning model [35, 30]. For example,
TPOT444https://github.com/rhiever/tpot is an early prototype that uses GP to
optimize a sequence of scikit-learn operations for both classification and
regression problems [27, 26, 25], and has been shown to work quite well across
a broad range of application domains ranging from epidemiological studies to
image classification to time series prediction [23]. Given the general design
of TPOT, the operations it optimizes over can be specialized for particular
problem domains. As another example, the TPOT-MDR project [33] showed that
TPOT can be specialized for genome-wide association studies (GWAS), and it
outperforms several state-of-the-art modeling methods on both simulated and
real-world GWAS problems because it considers a broad range of operations in
with one another. As such, we view GP as a strong candidate for a future
version of the PennAI Artificial Intelligence Engine, where the GP is seeded
with the best known algorithm configurations and uses the core principles of
GP (inheritance, mutation, and crossover)—distributed over a high-performance
computing cluster—to improve the algorithm configurations from there. This
brand of GP-based AI system would be useful for automatically launching new
analyses, but less useful for recommending particular algorithm configurations
to the user because GP does not provide a notion of the “next best” solution
to attempt.

Another offshoot of PennAI we are currently investigating is the use of a meta
genetic algorithm to find parameters (population size, generation count, etc.)
for a GP instance that work well, i.e., solve a given problem [32]. This
meshes well with the idea that the AI of PennAI will aid non-machine-learning
experts run complex algorithms, such as GP, without having to find or even
understand every single parameter.

Ultimately, PennAI will likely be comprised of several disparate AI algorithms
that use meta-data and meta-learning to improve the user experience and user
productivity by suggesting machine learning algorithms and parameters, as well
as providing other insights. As a result, we will be able to harness ensemble
techniques to collate the advice given by the numerous AI algorithms.

The time is now to bring AI technology to anyone that wants to use it for big
data analytics. The software and hardware technology exists and data has never
been bigger, more complex, and more plentiful. PennAI will provide both
machine learning and AI capability to both naive and expert users alike with a
user-friendly web and smartphone-enabled interface. We see AI technology such
as PennAI not as a replacement for the data scientist but rather as a data
science assistant that can suggest analyses to the user or provide
automatically generated results that are informed by previous analyses across
different data sets. The user can take these results as-is or use them as
inspiration in manual analyses. The democratization of AI is here.

##  10 Acknowledgements

This work was generously funded provided by the Perelman School of Medicine
and the University of Pennsylvania Health System of the University of
Pennsylvania. Additional funding was provided by National Institutes of Health
grants AI116794, DK112217, ES013508, and TR001878.

## References

  * [1] Arnaldo, I., Veeramachaneni, K., Song, A., O’Reilly, U.M.: Bring your own learner: A cloud-based, data-parallel commons for machine learning.  IEEE Computational Intelligence Magazine 10(1), 20–32 (2015)
  * [2] Buchanan, B., Shortliffe, E.: Rule-based expert systems: the MYCIN experiments of the Stanford Heuristic Programming Project (1984)
  * [3] Chodorow, K., Dirolf, M.: MongoDB: The Definitive Guide, 1st edn.  O’Reilly Media, Inc. (2010)
  * [4] Demšar, J., Curk, T., Erjavec, A., Črt Gorup, Hočevar, T., Milutinovič, M., Možina, M., Polajnar, M., Toplak, M., Starič, A., Štajdohar, M., Umek, L., Žagar, L., Žbontar, J., Žitnik, M., Zupan, B.: Orange: Data mining toolbox in python.  Journal of Machine Learning Research 14, 2349–2353 (2013)
  * [5] Ferrucci, D.A.: Introduction to “This is Watson”.  IBM Journal of Research and Development 56(3.4), 1–1 (2012)
  * [6] Goodfellow, I., Bengio, Y., Courville, A.: Deep learning.  MIT Press (2016)
  * [7] Hastie, T.J., Tibshirani, R.J., Friedman, J.H.: The Elements of Statistical Learning: Data Mining, Inference, and Prediction.  Springer, New York, NY, USA (2009)
  * [8] Kalousis, A.: Algorithm selection via meta-learning.  Ph.D. thesis, Universite de Geneve (2002)
  * [9] Kannappan, K., Spector, L., Sipper, M., Helmuth, T., La Cava, W., Wisdom, J., Bernstein, O.: Analyzing a decade of human-competitive (“humie”) winners: What can we learn?  In: Genetic Programming Theory and Practice XII, pp. 149–166. Springer International Publishing (2015)
  * [10] Koza, J.R.: Genetic programming: on the programming of computers by means of natural selection, vol. 1.  MIT Press (1992)
  * [11] La Cava, W., Danai, K., Spector, L.: Inference of compact nonlinear dynamic models by epigenetic local search.  Engineering Applications of Artificial Intelligence 55, 292–306 (2016)
  * [12] La Cava, W., Danai, K., Spector, L., Fleming, P., Wright, A., Lackner, M.: Automatic identification of wind turbine models using evolutionary multiobjective optimization.  Renewable Energy 87, 892–902 (2016)
  * [13] La Cava, W., Moore, J.: A general feature engineering wrapper for machine learning using $\epsilon$-lexicase survival.  In: European Conference on Genetic Programming, pp. 80–95. Springer (2017)
  * [14] La Cava, W., Moore, J.H.: Ensemble representation learning: an analysis of fitness and survival for wrapper-based genetic programming methods.  In: GECCO ’17: Proceedings of the Conference on Genetic and Evolutionary Computation. ACM (2017)
  * [15] La Cava, W., Silva, S., Vanneschi, L., Spector, L., Moore, J.: Genetic programming representations for multi-dimensional feature learning in biomedical classification.  In: European Conference on the Applications of Evolutionary Computation, pp. 158–173. Springer (2017)
  * [16] Langley, P.: Lessons for the Computational Discovery of Scientific Knowledge (2002)
  * [17] Moore, J.H., Andrews, P.C., Barney, N., White, B.C.: Development and evaluation of an open-ended computational evolution system for the genetic analysis of susceptibility to common human diseases.  In: European Conference on Evolutionary Computation, Machine Learning and Data Mining in Bioinformatics, pp. 129–140. Springer (2008)
  * [18] Moore, J.H., Greene, C.S., Hill, D.P.: Identification of Novel Genetic Models of Glaucoma Using the “EMERGENT” Genetic Programming-Based Artificial Intelligence System, pp. 17–35.  Springer International Publishing, Cham (2015).  DOI 10.1007/978-3-319-16030-6˙2
  * [19] Moore, J.H., Greene, C.S., Hill, D.P.: Identification of novel genetic models of glaucoma using the “emergent” genetic programming-based artificial intelligence system.  In: Genetic Programming Theory and Practice XII, pp. 17–35. Springer (2015)
  * [20] Moore, J.H., Hill, D.P., Fisher, J.M., Lavender, N., Kidd, L.C.: Human-Computer Interaction in a Computational Evolution System for the Genetic Analysis of Cancer, pp. 153–171.  Springer New York, New York, NY (2011).  DOI 10.1007/978-1-4614-1770-5˙9
  * [21] Moore, J.H., Hill, D.P., Saykin, A., Shen, L.: Exploring Interestingness in a Computational Evolution System for the Genome-Wide Genetic Analysis of Alzheimer’s Disease, pp. 31–45.  Springer New York, New York, NY (2014).  DOI 10.1007/978-1-4939-0375-7˙2
  * [22] Moore, J.H., White, B.C.: Genome-wide genetic analysis using genetic programming: The critical need for expert knowledge.  In: Genetic Programming Theory and Practice IV, pp. 11–28. Springer (2007)
  * [23] Olson, R.S., Bartley, N., Urbanowicz, R.J., Moore, J.H.: Evaluation of a Tree-based Pipeline Optimization Tool for Automating Data Science.  In: GECCO 2016, GECCO ’16, pp. 485–492. ACM, New York, NY, USA (2016).  DOI 10.1145/2908812.2908918
  * [24] Olson, R.S., La Cava, W., Orzeshowski, P., Urbanowicz Ryan J Moore, J.H.: Pmlb: A large benchmark suite for machine learning evaluation and comparison.  arXiv e-print. https://arxiv.org/abs/1703.00512 (2017)
  * [25] Olson, R.S., Moore, J.H.: Identifying and Harnessing the Building Blocks of Machine Learning Pipelines for Sensible Initialization of a Data Science Automation Tool.  arXiv e-print. http://arxiv.org/abs/1607.08878 (2016)
  * [26] Olson, R.S., Moore, J.H.: TPOT: A Tree-based Pipeline Optimization Tool for Automating Machine Learning.  JMLR 64, 66–74 (2016)
  * [27] Olson, R.S., Urbanowicz, R.J., Andrews, P.C., Lavender, N.A., Kidd, L.C., Moore, J.H.: Applications of Evolutionary Computation: 19th European Conference, EvoApplications 2016, Porto, Portugal, March 30–April 1, 2016, Proceedings, Part I, chap. Automating Biomedical Data Science Through Tree-Based Pipeline Optimization, pp. 123–137.  Springer International Publishing (2016).  DOI 10.1007/978-3-319-31204-0˙9
  * [28] Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al.: Scikit-learn: Machine learning in Python.  Journal of Machine Learning Research 12, 2825–2830 (2011)
  * [29] Ronald, E.M., Sipper, M., Capcarrère, M.S.: Design, observation, surprise! a test of emergence.  Artificial Life 5(3), 225–239 (1999)
  * [30] de Sá, A.G., Pinto, W.J.G., Oliveira, L.O.V., Pappa, G.L.: RECIPE: A Grammar-Based Framework for Automatically Evolving Classification Pipelines.  In: European Conference on Genetic Programming, pp. 246–261. Springer (2017)
  * [31] Silva, S., Muñoz, L., Trujillo, L., Ingalalli, V., Castelli, M., Vanneschi, L.: Multiclass classification through multidimensional clustering.  In: Genetic Programming Theory and Practice XIII, pp. 219–239. Springer (2016)
  * [32] Sipper, M., Fu, W., Ahuja, K., Moore, J.H.: From MEGATON to RASCAL: Surfing the parameter space of genetic programming (2017).  (in preparation)
  * [33] Sohn, A., Olson, R.S., Moore, J.H.: Toward the automated analysis of complex diseases in genome-wide association studies using genetic programming.  arXiv e-print. https://arxiv.org/abs/1702.01780 (2017)
  * [34] Vanneschi, L., Archetti, F., Castelli, M., Giordani, I.: Classification of oncologic data with genetic programming.  Journal of Artificial Evolution and Applications 2009, 6 (2009)
  * [35] Zutty, J., Long, D., Adams, H., Bennett, G., Baxter, C.: Multiple objective vector-based genetic programming using human-derived primitives.  In: Proceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation, pp. 1127–1134. ACM (2015)


# Calculus of Consent via MARL: Legitimating the Collaborative Governance
Supplying Public Goods

Yang Hu
Tsinghua University
Beijing, China
<EMAIL_ADDRESS>
&Zhui Zhu
Tsinghua University
Beijing, China
<EMAIL_ADDRESS>
&Sirui Song
McGill University
Montréal, Canada
<EMAIL_ADDRESS>
&Xue Liu
McGill University
Montréal, Canada
<EMAIL_ADDRESS>
&Yang Yu
Tsinghua University
Beijing, China
<EMAIL_ADDRESS>

###### Abstract

Public policies that supply public goods, especially those involve
collaboration by limiting individual liberty, always give rise to
controversies over governance legitimacy. Multi-Agent Reinforcement Learning
(MARL) methods are appropriate for supporting the legitimacy of the public
policies that supply public goods at the cost of individual interests. Among
these policies, the inter-regional collaborative pandemic control is a
prominent example, which has become much more important for an increasingly
inter-connected world facing a global pandemic like COVID-19. Different
patterns of collaborative strategies have been observed among different
systems of regions, yet it lacks an analytical process to reason for the
legitimacy of those strategies. In this paper, we use the inter-regional
collaboration for pandemic control as an example to demonstrate the necessity
of MARL in reasoning, and thereby legitimizing policies enforcing such inter-
regional collaboration. Experimental results in an exemplary environment show
that our MARL approach is able to demonstrate the effectiveness and necessity
of restrictions on individual liberty for collaborative supply of public
goods. Different optimal policies are learned by our MARL agents under
different collaboration levels, which change in an interpretable pattern of
collaboration that helps to balance the losses suffered by regions of
different types, and consequently promotes the overall welfare. Meanwhile,
policies learned with higher collaboration levels yield higher global rewards,
which illustrates the benefit of, and thus provides a novel justification for
the legitimacy of, promoting inter-regional collaboration. Therefore, our
method shows the capability of MARL in computationally modeling and supporting
the theory of calculus of consent, developed by Nobel Prize winner J. M.
Buchanan.

##  1 Introduction

As brilliantly revealed in J. M. Buchanan’s celebrated book The Calculus of
Consent [1], collective actions are always formed by (probably conflicting)
individual actions, during which process some individuals should rationally
alienate or waive their rights of liberty based on the calculus of consent.
This outstanding theory of public choice applies to the supply of public
goods, which generally requires the collaboration among multiple parties that
could restrict the liberty of some parties for the sake of public interest.
Thus, the controversy between the extent of liberty restriction and the
necessary level of collaboration is naturally raised for debate, and governors
will face the problem of legitimacy whenever they fail to explain the
necessity and intensity of collaborative policies.

It should be noticed that the making of optimal policies and their legitimacy
explanations involve delicate quantified calculation of individual and public
interests, or more specifically in the scope of public goods supply, the
balance of costs and benefits. The calculation is of such high computational
complexity that is beyond the capability of human instincts, which impairs the
legitimacy and credibility of policy-makers, and thus restricts them from
fully functioning to pursue the greatest good for the public. Fortunately, the
rapidly developing computational tools for decision making, such as optimal
control and reinforcement learning, have enabled us to find optimal solutions
for complicated problems, so they can effectively serve as convincing
justifications for policy-makers.

There is no doubt that the most recent large-scale debate over the supply of
public goods is about COVID-19 pandemic control policies. In the highly
globalized and interconnected world today, pandemic outbreaks have proven to
be more devastating to human society than any time in history — they not only
cause millions of deaths around the world, but also jeopardize the global
value chain (GVC) [2] and give rise to billions of dollars of economic losses.
Economists have realized that the control of infectious diseases falls into
the category of global public goods (GPGs) that cannot be efficiently supplied
without intensive inter-regional collaboration [3]. And here naturally comes
the problem of legitimacy: how can a central government convince its local
governments, or an international association convince its member countries,
that enforcing strict lockdown policies is an appropriate and necessary
measure to take? Why should the latter be willing to carry out those policies
that might harm their own economic interests?

Indeed, regions are not equally willing to collaborate at the cost of their
own benefits — regions in a collaborative system are expected to enforce
strict lockdown policies to guarantee global well-being, even if they will
damage the local economy; regions in a self-centered system, however, will
more likely carry out loose restrictions to save the local economy, at the
cost of wider-spread diseases and global utility losses which they do not
really care. As a result, different behavior patterns will be observed in
systems of regions that vary greatly in the the willingness to collaborate,
and thus a legitimate policy in one system might not be so in another — strict
lockdown policies can be easily enforced in unitary governments, but harder in
federal governments; vaccines can be efficiently distributed within a country,
but significantly less smoothly within an international association.
Differences in the willingness to collaborate root in the difference in social
capitals, which plays a decisive role in the organization of collaborative
supply of public goods.

Such “willingness” should be emphasized in the analysis of inter-regional
pandemic control policies. It can be quantitatively characterized by the
proportion of global utility appearing in each region’s policy-making
objective, which we shall refer to as collaboration levels. Differences in
collaboration levels may root from different sources — it could be the natural
consequence of a coherent and unified political structure (e.g., as in China)
that always acts in pursuit of greatest global utility; or it could be due to
the political relationship and economic inter-dependence (e.g., as in the EU),
so that the high externality of pandemic control prompts the regions to care
more about the global utility.

Finding effective pandemic control measures involves both epidemiological and
socioeconomic considerations. On the epidemiological side, research has
revealed the effect of human mobility on the COVID-19 pandemic [4, 5],
pointing out the fact that respiratory infectious diseases like the COVID-19
can hardly be controlled without strict international restrictions on the
mobility of population and commodities. However, in the socioeconomic
perspective, strict lockdown policies will probably damage local economy [6],
leading to unfavorable consequences like unemployment, supply shortage or even
depression. Therefore, policy-makers face an inherent trade-off between
pandemic control and economic well-being, and the equilibrium is determined by
the pandemic-tolerance level (whether it has abundant medical resources to
endure a long-term pandemic outburst) and the lockdown-tolerance level
(whether its economic structure can survive a long-term lockdown).

Computational approaches are playing an increasingly important role in search
of optimal pandemic control policies. For example, Tsay et al. proposed an
optimization-based decision-making framework to calculate optimal pandemic
control policies [7]. Artificial intelligence tools such as Multi-Agent
Reinforcement Learning (MARL) algorithms have also be employed to allow more
complex models, balancing health and economic costs [8, 9]. However, all of
these studies focus on centralized control policies only, so that they do not
take into consideration how multiple independent regions should and would
collaborate to fight against the pandemic. Therefore, we feel motivated to
invent new computational models that helps to analyze the underlying patterns
of inter-regional collaboration, which also helps to find and legitimize
optimal pandemic control policies.

The structure of the multi-region collaborative pandemic control problem bears
several challenges, such as intensive coupling among regions brought by
pandemic spread, the exponential explosion nature of pandemic spread, and the
instability of observed environment due to inderdependency of regional
policies. To address these inherent challenges, we design an MARL model called
Intelligent Region Collaboration (IRC), where each region is represented by a
learning agent, and each agent’s action is to block a proportion of traffic
from any other regions into it, which abstracts the mobility restriction
policy in reality. Regions are assigned different pandemic-tolerance levels
and lockdown-tolerance levels to capture the major factors that influence
policy-making. To reflect different collaboration levels, each agent is
assumed to receive both an individual reward and a global reward, and the
trade-off weight between these two terms, known as the reward mixing ratio,
quantifies collaboration levels. The MARL model is trained with a specifically
designed actor-critic algorithm that “decouples” the highly unstable
environment faced by each agent into a stable local environment learned by the
agent itself, and a stable global environment learned globally.

The IRC method proposed in this paper serves as an effective tool to find
optimal pandemic control policies under different inter-regional collaboration
levels. These policies not only help policy-makers in the real world, but also
provide a perspective to explore the influence of different tolerance levels
and collaboration levels on agents’ behavioral patterns and the final
outcomes. For this purpose, we construct an exemplary environment with agents
of different tolerance levels, and observe the policy learned by our model
under different collaboration levels. The learned policies are evaluated by
multiple metrics, compared with two heuristic baseline policies. Experimental
results show that different collaboration patterns appear under different
collaboration levels in an interpretable way. It is observed that, when a
region only cares about its own interest, its optimal pandemic control policy
is almost completely based on its own tolerance levels, and thus the amount of
blocked traffic is highly imbalanced among different regional types; on the
other hand, as global interests become dominant in a region’s reward
composition, it is more likely for a region to sacrifice a small portion of
its local reward by blocking extra amount of traffic into it, in exchange for
lower lockdown penalties of more vulnerable regions. The collaboration is also
characterized by more evenly distributed mobility among different types of
regions.

The contribution of this paper is that we design an MARL model to find optimal
region-collaboration pandemic control policies, which contains an adjustable
parameter that quantifies the collaboration level among regions, or more
specifically, the proportion of global reward in each agent’s mixed reward. In
this way, we obtain a new perspective to analyze the collaboration behavior of
regions with different types under different collaboration levels, and by
examining the patterns in the learned optimal policies we can better
understand the underlying logic of inter-regional collaboration, and thus
provide new justifications for the legitimacy of collaborative governance
supplying public goods.

##  2 Related Works

In this section, we discuss the research that is the most pertinent to the
work in this paper.

###  2.1 Multi-Agent Reinforcement Learning

Reinforcement learning (RL) has long been an active research field for many
decision-making problems [10, 11]. Despite its success in single-agent
settings, Multi-Agent Reinforcement Learning (MARL) that deals with multiple
autonomous agents is much more complicated, and has seen a rapid development
in the past decade [12, 13]. State-of-the-art MARL algorithms are able to show
human-level performance in complex multi-agent games such as DOTA [14] and
StarCraft II [15].

In the typical cooperative MARL setting, the environment responds to the
aggregate action of all agents with a single global reward, and agents take
joint efforts to maximize total global reward. Therefore, a natural idea is to
learn the decomposition of the global reward first, and then apply value-based
RL algorithms for each agent, which is adopted by popular algorithms like VDN
[16], QMIX [17], QTRAN [18], etc. Another approach is to adapt policy gradient
methods for multi-agent settings, among which the most successful algorithms
are MADDPG [19], COMA [20] and MAAC [21]. There is also much research
investigating the cooperative behaviour of MARL agents in various environments
[22, 23].

The multi-agent setting in this paper is different from the typical
cooperative setting in most existing literature. Our pandemic control task is
special in that agents are allowed to retrieve their own local rewards in
response to their actions, and thus the value decomposition network is
unnecessary. Meanwhile, collaboration level is abstracted as an adjustable
hyperparameter, so that collaborative behavior can be controlled rather than
only observed.

###  2.2 Pandemic Control

In epidemiological studies, the Susceptible-Infected-Recovered (SIR) model is
a widely used mathematical model for pandemic transmission, on which many
research on pandemic control strategies are based [24]. It is also modified
for more realistic modelling: SEIR model added an Exposed state to make the
model more generalizable [25], while DURLECA proposes the SIHR model to
capture the challenges posed by asymptomatic infections [8].

Based on these pandemic transmission models, researchers have discussed how to
find the optimal pandemic control strategy in different settings using
traditional optimization approaches [26, 27, 28]. More recently, the
application of RL methods is proposed, where the actions of all regions are
determined by one agent in a centralized manner [8]. Compared with these
existing studies, our IRC method is the first to apply MARL methods for
optimal pandemic control policies, and to investigate the behaviour patterns
of different regions under different collaboration levels.

##  3 Modeling and Method

###  3.1 Environment Modeling

The environment model focuses on the traffic (or mobility) and pandemic
transmission among different regions. Therefore, each region can be simply
regarded as a single node within a fully connected graph, and all within-node
details are excluded from our model.

Mobility. The amount of mobility demand $M_{\text{d},i,j}^{t}$ between each
pair of regions $i,j$ at each time step is pre-determined. Each region
regulates its pandemic control policy by determining the proportion of
incoming mobility demand that can be fulfilled (note that traffic is
controlled by destination region rather than the departure region). Therefore,
region $j$ is allowed to determine $p_{i,j}^{t}\in[0,1]$ for all $i\neq j$,
and fulfills only that proportion of the total mobility demand. A more
rigorous formulation of mobility demand and actions can be found in Appendix
A.1.

It is reasonable to abstract pandemic control policy as incoming mobility
restrictions. Mobility between regions reflects the quantity of economic
activities, and mobility restrictions will lead to losses of economic
interests; mobility of people and commodities can be easily controlled by
region governments via border and public transportation controls; mobility is
also closely related to pandemic control as unrecognized infections will
spread the virus to the destination. Therefore, in our model, agents are
expected to maximize the fulfillment of mobility demands, but also wisely
restrict mobility to reduce the spread of pandemic simultaneously.

Pandemic transmission. The pandemic transmission model is based on the
Susceptible-Infected-Hospitalized-Recovered (SIHR) model proposed in [8]. The
pandemic state of region $i$ at time step $t$ is
$E^{t}_{i}=(S^{t}_{i},I_{i}^{t},H_{i}^{t},R_{i}^{t})$, which consists of the
number of susceptible, infected, hospitalized, and recovered population within
that region. Agents are able to observe the number of hopitalization and
recovery, but cannot distinguish between susceptible (healthy) people and
infected people that are both asymptomatic. Therefore, the visible pandemic
state of region $i$ at time step $t$ is
$E^{t}_{\text{v},i}=(S^{t}_{i}+I_{i}^{t},H_{i}^{t},R_{i}^{t})$. The detailed
model for pandemic transmission is deferred to Appendix A.2.

###  3.2 Problem Formulation

The target of our multi-region collaborative pandemic control problem is two-
fold — agents are expected to minimize the spread of infection, and fulfill
the maximum proportion of mobility demand at the same time. Therefore, the
local reward $R_{i}$ of region $i$ is determined by two factors: how much
mobility demand into this region is fulfilled, and how serious the pandemic
state is in this region. More rigorously, each region $i$ receive a local
reward $R_{i}$ for its pandemic situation at each time step, which is a sum of
the pandemic-spread cost $C_{\text{p},i}$ and the mobility-control cost
$C_{\text{m},i}$

$R_{i}=-C^{t}_{\text{p},i}-C^{t}_{\text{m},i},$ (1)

The global reward $R_{\text{g}}$ (the common interest of all regions) is the
sum of all local rewards

$R_{\text{g}}=\sum_{i=1}^{n}R_{i}=\sum_{i=1}^{n}\left(-C^{t}_{\text{p},i}-C^{t}_{\text{m},i}\right).$
(2)

Here costs $C^{t}_{\text{p},i}$ and $C^{t}_{\text{m},i}$ grow exponentially
with regard to the number of hospitalized patients ($H^{t}_{i}$) and an
accumulated amount of blocked mobility, respectively. Two characteristic
hyperparameters, namely pandemic-tolerance level $H_{0,i}$ and lockdown-
tolerance level $L_{0,i}$, represent the exponential grow rate mentioned
above. The pair $(H_{0,i},L_{0,i})$ is also referred to as region types since
they sufficiently reflect a region’s resilience for pandemic outbreak and
lockdown — the higher the hyperparameter, the more resilient a region is.
Detailed definitions of these costs and tolerance levels are deferred to
Appendix A.3.

Although the two tolerance levels seem highly abstract and artificial, they
are meaningful in practice. Countries and cities have different supply of
medical resources, and those with more medical resources tend to be more
resilient to a pandemic outbreak. Meanwhile, regions whose economy rely
heavily on travellers and imported commodities will be more reluctant to carry
out complete lockdown policy, as it may hurt the economy even more severely
than the pandemic outbreak. Therefore, the simple abstraction does capture the
main conflicting factors in pandemic control.

Now we can follow the multi-region collaborative pandemic control problem. The
objective of each agent is a weighed sum of local and global rewards, namely

$p^{t}_{i}=\mathop{\arg\max}_{p}\big{\\{}R_{i}(p;M^{t}_{\text{d},i},E^{t}_{i})+\alpha
R_{\text{g}}(p;M^{t}_{\text{d}},E^{t})\big{\\}}.$ (3)

where we introduce a reward mixing ratio $\alpha$ to represent the proportion
that the global reward accounts for in the mixed reward
$\tilde{R}_{i}=R_{i}+\alpha R_{\text{g}}$.

###  3.3 Intelligent Region Collaboration (IRC)

Figure 1: Schematic diagram of Intelligent Region Collaboration (IRC) model.

To solve the multi-region collaborative pandemic control problem, we design an
MARL method called Intelligent Region Collaboration (IRC). The IRC method
implements a multi-agent actor-critic RL algorithm enhanced by GNN networks.
Each region is modeled as an RL agent. For agent $i$ at time step $t$, it
first acquires an observation consisting of the visible pandemic state
$E_{\text{v}}^{t}$ and mobility demands
$(M^{t}_{\text{d}},M^{t+1}_{\text{d}},M^{t+2}_{\text{d}},M^{t+3}_{\text{d}})$
for a period of 4 time steps from the environment; then based on this
information, the agent determines its action $p^{t:(t+3)}_{i}$ for the next 4
time steps. Note that control actions are carried out every several time
steps, which makes practical sense since changes in mobility demands and
control policies can only be applied periodically in reality.

Details of our RL framework are presented below:

  * •

State. The state includes the visible pandemic state $E_{\text{v}}^{t}$ and
its temporal derivative $\nabla_{t}E_{\text{v}}^{t}$.

  * •

Action. The action of each RL agent $i$ is a column vector
$p^{t}_{i}=(p^{t}_{1,i},\cdots,p^{t}_{n,i})^{\top}$ ($n$ is the number of
regions), where $p^{t}_{j,i}$ is the fulfilled proportion of mobility demand
$M^{t}_{\text{d},j,i}$.

  * •

Reward. Agent $i$ receives a mixed reward $\tilde{R}_{i}=R_{i}+\alpha
R_{\text{g}}$ as defined previously.

  * •

Learning Algorithm. The IRC method is based on the GNN-enhanced Deep
Deterministic Policy Gradient (DDPG) method used in DURLECA [8]. The GNN is
used to estimate the future pandemic status from the observations, which can
utilize the underlying graph structure. Each agent is trained by the DDPG
algorithm introduced in [29].

To deal with the challenge of mutually coupled and volatile environment
observed by each agent that obstructs efficient training, critics in our IRC
model are specially designed to utilize physical decomposablity of local and
global rewards in our setting. Therefore, instead of employing a single critic
for each agent and introducing an extra value-decomposition network, as in
standard MADDPG agents [19], we employ an individual local critic for each
agent and an additional shared global critic for all agents to learn local and
global rewards separately. More specifically, each DDPG agent in our IRC
method is equipped with an actor and a local critic, and has access to a
shared global critic. The local critic is customized for each agent, which
learns the $Q$-function associated with the local reward $R_{i}$; the global
critic is shared by all agents, which learns the $Q$-function associated with
the global reward $R_{\text{g}}$. The actor generates actions that maximizes
the weighed sum of the local critic output and the global critic output. In
this way, the highly unstable environment faced by each agent is decoupled
into a stable local environment learned by the agent itself and a stable
global environment learned by all the agents from a global view.

##  4 Experiment and Evaluations

In this section, we present the experimental results in a representative
setting, in order to reveal the patterns of collaboration behavior among
regions with different types under different collaboration levels, based on
the optimal policies learned by our IRC method. We first describe the setup
and evaluation metrics used in the experiments, and then present the results
and our interpretations.

###  4.1 Experimental Settings

The experiment focuses on the early “outbreak” stage of pandemic transmission,
where most regions contain few infected people, but one specific source region
(labelled as region #0) is suffering from an already out-of-control pandemic
outbreak ($R_{0}>1$), and infections will spread from the source region to
other regions. This stage is critical to successful pandemic control. Since we
care more about preventing the vast pandemic spread among all the regions than
controlling infections in the source region (which is configured to be
impossible, as $R_{0}>1$ in region #0), we specially exclude the pandemic
penalty from region #0’s local reward.

Environment and region types. With the main purpose of revealing the influence
of tolerance levels and collaboration level on the outcome, the artificial
setting contains only 5 regions. Each region has an initial population of
10,000,000 people, a daily traffic of 5,000 people for each travelling route,
and 2,000 initial infections in source region #0. Apart from the source
region, other regions (#1 through #4) are designed to demonstrate different
strategies learned by agents of different region types. Here we assume the
tolerance levels are set to be either “high” ($H_{+}$ or $L_{+}$) or “low”
($H_{-}$ or $L_{-}$), where $H_{+}>H_{-}$ and $L_{+}>L_{-}$. It is evident
that 4 non-source regions are sufficient to cover all possible region types,
namely $(H_{+},L_{+}),(H_{+},L_{-}),(H_{-},L_{+}),(H_{-},L_{-})$,
respectively.

We point out that region types are abstract hyperparameters specific to our
pandemic model, so that it is their relative relationship rather than absolute
values that matter in our illustrative experiment, which justifies the two-
level setting. For the sake of implementation, we assign the specific values
as $H_{+}=0.003$, $H_{-}=0.001$, $L_{+}=72$ and $L_{-}=24$. The source region
#0 is special in that its pandemic situation is not concerned, so we assign to
it a lockdown-tolerance level $L_{0,0}=0.05$.

Collaboration levels. To investigate agents’ behavioral patterns under
different collaboration levels, our IRC models are trained with 3 different
reward mixing ratios $\alpha\in\\{0.01,0.40,10.0\\}$. For each configuration,
the training procedure is repeated for 10 times with different random seeds,
and the model that yields the highest reward is selected for further
evaluation and analysis.

###  4.2 Evaluation Protocols

Baselines. Our MARL models are compared against the following two baselines
(or conceptually “expert policies”), which are abstracted from common and most
easily implementable practices.

  * •

Fixed policy. Each region allows a fixed proportion $p_{\text{fix}}$ of all
incoming traffic for entrance. The proportion is global in the sense that it
works for all regions and throughout the pandemic period. This abstracts
typical control strategies like permanently restricting international flights.

  * •

Threshold policy. This is a more flexible strategy that dynamically “respond
to” the changing pandemic situation, where lockdown is enforced when there are
many observed (i.e., hospitalized) patients, indicating that the pandemic
situation is worsening, and the cumulative mobility loss is still acceptable,
indicating that the lockdown will not hurt economy too seriously. More
specifically, the policy sets each action $p^{t}_{i,j}$ by the rule

$p^{t}_{i,j}=\begin{cases}0&H^{t}_{i}>H_{\text{th}}~{}\text{and}~{}L^{t}_{i}<L_{\text{th}}\\\
1&\textrm{otherwise}\end{cases}.$ (4)

Metrics. Several metrics reflecting the effectiveness of pandemic control and
the extent of collaboration are used to quantitatively evaluate the
performance of our models against the baselines:

  * •

Mean global reward $\overline{R}_{\text{g}}$. This is a direct revelation of
the performance that the model is trained to optimize. Here we only need the
mean of global reward, since the local rewards of all agents sum to the global
reward.

  * •

The mean $\overline{H}$ and maximum $H_{\text{max}}$ number of total
hospitalized patients. These two metrics represent the pressure on local
medical services brought by the pandemic control policy, which reflect the
feasibility of the pandemic control policy in medical sense.

  * •

Mean action $\overline{p}$. Mean action is defined as the ratio of total
actual mobility over total mobility demand into a region. It represents the
strictness of inter-regional lockdown enforced by a region, and acts as an
indicator for the economic loss induced by the pandemic control policy.

  * •

Type-wise analysis. To analyze the collaboration behavior of agents with
different regional types, the mean hospitalization $\overline{H}$ and mean
mobility $\overline{p}$ within regions of the same type are also calculated,
and presented in the form of 2-by-2 matrices and/or radar plots.

###  4.3 Experimental Results

The results of baseline polices and our IRC models are shown in Table 1. It is
clear that, regardless of the mixing ratio $\alpha$, our IRC models outperform
both baselines by receiving larger global rewards, higher mobility and lower
mean hospitalization rate. Therefore, our IRC models achieve better
performance at balancing the pandemic cost and lockdown cost of each region,
and at balancing the demand of different regions.

Table 1: The performance of different models. model | parameter | $\bm{\overline{R}_{\text{g}}}$ | $\bm{\overline{H}}$ | $\bm{H_{\textbf{max}}}$ | $\bm{\overline{p}}$
---|---|---|---|---|---
Fixed | $p_{\text{fix}}=0.5$ | $64.16$ | $88.29$ | $276.68$ | $0.5000$
Threshold | $H_{\text{th}}=1,L_{\text{th}}=168$ | $88.82$ | $92.65$ | $202.17$ | $0.8198$
IRC | $\alpha=0.01$ | $\bm{103.89}$ | $59.46$ | $373.25$ | $0.7753$
IRC | $\alpha=0.40$ | $\bm{106.97}$ | $31.29$ | $327.90$ | $0.7090$
IRC | $\alpha=10.0$ | $\bm{108.13}$ | $27.21$ | $232.73$ | $0.6909$

$\alpha=0.01$

$\alpha=0.40$

$\alpha=10.0$

(a) radar plot of $\overline{p}$.

Legend:

$\alpha=0.01$

$\alpha=0.40$

$\alpha=10.0$

(b) temporal change of $\overline{p}$. Figure 2: Type-wise analysis of mean
action $\overline{p}$ for different mixing ratios.

Despite the simplicity of the setting, this environment clearly illustrates
different behavior patterns of agents of different regional types. For agents
trained under different mixing ratios, the type-wise analysis of mean actions
and mean hospitalization rates are shown in Figure 2 and 3. It is observed
that, when agents make decisions based largely on local rewards (i.e., in the
case of $\alpha=0.01$), they tend to select actions that best serve their own
interests — the $(H_{-},L_{+})$-region is extremely vulnerable to outbursts of
infections, so its optimal policy is to enforce strict lockdown policies; the
$(H_{+},L_{-})$-region cannot afford long-term lockdowns, so it prefers to
largely open up inward traffic; the $(H_{-},L_{-})$-region suffers from both a
pandemic outbreak and a large cumulative mobility loss, so it strictly blocks
all inward traffic initially, but is forced to open up later to avoid
cumulative lockdown penalty. This is clearly shown in the radar plot (Figure
2(a)), where the colored area (representing mobility) leans towards
$(H_{+},L_{-})$-direction, indicating that regions that can afford more
patients but not strict lockdowns are more likely to open up.

The above behavioral patterns are similar to what we have observed in the very
early stage of COVID-19 outbreak, where regional governments have not
established unified collaboration protocols. The $(H_{-},L_{+})$-region
corresponds to those regions in reality that have limited medical resources
and relatively self-sufficient economies, so that they are willing to avoid
local panic for medical resources at the cost of economic losses. The
$(H_{+},L_{-})$-region represents those regions with export-oriented economies
that rely heavily on inter-regional population and commodity mobility, so they
will not voluntarily enforce lockdown policies, but would rather maintain
economic vitality even at the cost of increasing number of infections. The
$(H_{-},L_{-})$-region, on the other hand, is a representative of under-
developed regions that are the most vulnerable to a pandemic outbreak —
without help from other regions, they will soon fall into the dilemma that
neither open-up nor lockdown is favorable.

However, as the global reward gains more weight in agents’ optimization
objectives, their behavior gradually changes. The agents with higher lockdown-
tolerance level ($L_{+}$) tend to share the lockdown burden with those agents
who have lower lockdown-tolerance level ($L_{-}$), as they enforce stricter
lockdown from the source region. Such changes will not be observed if each
agent focuses on its own interest only, since for those $H_{+}$-regions that
voluntarily share the burden, slightly larger infection numbers do not hurt
them as badly as the extra lockdown costs; however, if they choose not to
cooperate, the most vulnerable $(H_{-},L_{-})$-region has to block more
traffic from $H_{+}$-regions to avoid imported infections, which vastly
increases the lockdown cost of the $(H_{-},L_{-})$-region or even leads to
oscillations in action. Therefore, enforcing extra lockdown by those
$H_{+}$-regions against the source city helps more vulnerable cities to suffer
less from pandemic outbreaks and/or cumulative mobility losses, which is
indeed a kind of altruistic collaboration that promotes overall welfare and
realizes better inter-regional pandemic control. The trend of changing
behavior is illustrated in Figure 3, and is also clear from the radar plot
(Figure 2(a)), as the colored area becomes more balanced among different types
of regions.

The influence of the collaborative behavior of our IRC agents can also be
reflected by the mean hospitalization matrix, as illustrated in Figure 3. It
is observed that, as the mixing ratio $\alpha$ increases, the number of
hospitalized people in the $(H_{-},L_{-})$-region gradually decreases even
when its mobility increases at the same time. Meanwhile, the majority of
pandemic penalties are still undertaken by $H_{+}$-regions (since they still
allow most inward mobility), which have more medical resources and are thus
more resilient to such mild increases in infections. Therefore, our IRC agents
indeed learn to balance the demand of regions with different regional types,
and such balanced collaboration behavior is helpful to promote overall welfare
(indicated by increased $\overline{R}_{\text{g}}$ and lower $\overline{H}$).

Legend:

Figure 3: Type-wise trend of change for $\overline{p}$ and $\overline{H}$ when
mixing ratio increases.

As a short summary, our IRC models successfully learn different policies for
different mixing ratios, and the learned behavior shows an interpretable
changing pattern of collaboration behavior — regions with abundant medical
resources and high resilience for pandemic outbreak will voluntarily enforce
stricter lockdown policies to help those vulnerable regions at higher
collaboration levels. It is also worth mentioning that, as the mixing ratio
$\alpha$ increases, our IRC models are more likely to find policies that yield
higher global rewards, which demonstrates the benefit of collaboration, and
confirms the mixing ratio to be a good adjustable parameter representing the
collaboration level.

##  5 Conclusions and Future Work

In this paper, we analyze collaborative patterns of multi-region pandemic
control behavior under different collaboration levels, using the optimal
policies found by our specially-designed MARL method. We introduce a reward
mixing ratio to abstract the inter-regional collaboration level that can be
adjusted to reveal agents’ different behavior difference under different
collaboration levels. Experimental results in an exemplary environment show
that, as the global reward accounts for a larger portion of each agent’s
objective, the actions and rewards of regions of different types become more
balanced, and agents tend to learn policies that not merely promote their own
interests, but also collaborate to help reduce the loss of other regions. One
major take-away message here is that higher collaboration level will lead to
better and more balanced pandemic control policies, in that regions will
voluntarily help to coordinate the demands of different regions, and regions
more resilient to lockdown tend to sacrifice a mild portion of its own utility
to protect more vulnerable regions.

Our framework provides a novel computational perspective for understanding and
promoting collaboration for the supply of public goods. As a representative
type of them, pandemic control cannot be supplied with a few regions or
through a few isolated policies, while different attitudes towards inter-
regional collaboration enables different pandemic control policies, and will
possibly lead to different outcomes. This reminds us that, to better cope with
global catastrophes like COVID-19 and more efficiently provide public goods to
promote global welfare, international collaboration should be encouraged and
all related parties should be called for to actively assume international
responsibilities, so that the strong will protect the weak by sharing the
losses. The legitimacy of such collaborative policies that restrict individual
liberty can be justified by the optimality guarantee of the computational
methods. We believe this framework can be generalized to analyze the supply of
other public goods that involves collaboration.

The model of this paper is limited in that it regards inter-regional mobility
control as the only form of action. However, regions’ within-regional pandemic
control strategies will also induce epidemiological and economic costs, which
also potentially influences inter-region collaboration. For example, a
destination region may be more willing to accept inward mobility if the source
region enforces better control within itself. Therefore, a more comprehensive
modelling is left as future work.

## References

  * Buchanan and Gordon [1965] James M. Buchanan and Tullock Gordon.  _The Calculus of Consent: Logical Foundations of Constitutional Democracy_.  Ann Arbor: University of Michigan Press, 1965.
  * Kimura et al. [2020] Fukunari Kimura, Shandre Mugan Thangavelu, Dionisius Narjoko, and Christopher Findlay.  Pandemic (covid-19) policy, regional cooperation and the emerging global production networkâ€ .  _Asian Economic Journal_ , 34(1):3–27, 2020.
  * Brown and Susskind [2020] Gordon Brown and Daniel Susskind.  International cooperation during the COVID-19 pandemic.  _Oxford Review of Economic Policy_ , 36(Supplement-1):S64–S76, 09 2020.
  * Kraemer et al. [2020] M. U. G. Kraemer, C.-H. Yang, B. Gutierrez, C.-H. Wu, B. Klein, D. M. Pigott, L. Du Plessis, N. R. Faria, R. Li, W. P. Hanage, et al.  The effect of human mobility and control measures on the covid-19 epidemic in china.  _Science_ , 368(6490):493–497, 2020.
  * Badr et al. [2020] H. S. Badr, H. Du, M. Marshall, E. Dong, M. M. Squire, and L. M. Gardner.  Association between mobility patterns and covid-19 transmission in the usa: a mathematical modelling study.  _The Lancet Infectious Diseases_ , 20(11):1247–1254, 2020.
  * Barua [2020] Suborna Barua.  Understanding coronanomics: The economic implications of the coronavirus (covid-19) pandemic.  MPRA Paper 99693, University Library of Munich, Germany, April 2020.  URL https://ideas.repec.org/p/pra/mprapa/99693.html.
  * Tsay et al. [2020] C. Tsay, F. Lejarza, M. A. Stadtherr, and M. Baldea.  Modeling, state estimation, and optimal control for the us covid-19 outbreak.  _Scientific reports_ , 10(1):1–12, 2020.
  * Song et al. [2020] S. Song, Z. Zong, Y. Li, X. Liu, and Y. Yu.  Reinforced epidemic control: Saving both lives and economy.  _arXiv preprint arXiv:2008.01257_ , 2020.
  * Khadilkar et al. [2020] H. Khadilkar, T. Ganu, and D. P. Seetharam.  Optimising lockdown policies for epidemic control using reinforcement learning.  _Transactions of the Indian National Academy of Engineering_ , 5(2):129–132, 2020.
  * Kaelbling et al. [1996] L. P. Kaelbling, M. L. Littman, and A. W. Moore.  Reinforcement learning: A survey.  _Journal of artificial intelligence research_ , 4:237–285, 1996.
  * Sutton and Barto [2018] R. S. Sutton and A. G. Barto.  _Reinforcement learning: An introduction_.  MIT press, 2018.
  * Zhang et al. [2019] K. Zhang, Z. Yang, and T. Başar.  Multi-agent reinforcement learning: A selective overview of theories and algorithms.  _arXiv preprint arXiv:1911.10635_ , 2019.
  * Buşoniu et al. [2010] L. Buşoniu, R. Babuška, and B. de Schutter.  Multi-agent reinforcement learning: An overview.  _Innovations in multi-agent systems and applications-1_ , pages 183–221, 2010.
  * OpenAI [2018] OpenAI.  Openai five.  https://blog.openai.com/openai-five/, 2018.
  * Vinyals et al. [2017] O. Vinyals, T. Ewalds, S. Bartunov, P. Georgiev, A. S. Vezhnevets, M. Yeo, A. Makhzani, H. Küttler, J. Agapiou, J. Schrittwieser, et al.  Starcraft ii: A new challenge for reinforcement learning.  _arXiv preprint arXiv:1708.04782_ , 2017.
  * Sunehag et al. [2018] P. Sunehag, G. Lever, A. Gruslys, W. M. Czarnecki, V. Zambaldi, M. Jaderberg, M. Lanctot, N. Sonnerat, J. Z. Leibo, K. Tuyls, et al.  Value-decomposition networks for cooperative multi-agent learning based on team reward.  In _Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems_ , pages 2085–2087. International Foundation for Autonomous Agents and Multiagent Systems, 2018.
  * Rashid et al. [2018] T. Rashid, M. Samvelyan, C. S. Witt, G. Farquhar, J. Foerster, and S. Whiteson.  Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning.  In _International Conference on Machine Learning_ , pages 4292–4301, 2018.
  * Son et al. [2019] K. Son, D. Kim, W. J. Kang, D. E. Hostallero, and Y. Yi.  Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning.  In _International Conference on Machine Learning_ , pages 5887–5896, 2019.
  * Lowe et al. [2017] R. Lowe, Y. Wu, A. Tamar, J. Harb, P. Abbeel, and I. Mordatchr.  Multi-agent actor-critic for mixed cooperative-competitive environments.  _arXiv preprint arXiv:1706.02275_ , 2017.
  * Foerster et al. [2018] J. N. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson.  Counterfactual multi-agent policy gradients.  In _Thirty-Second AAAI Conference on Artificial Intelligence_ , 2018\.
  * Iqbal and Sha [2019] S. Iqbal and F. Sha.  Actor-attention-critic for multi-agent reinforcement learning.  In _International Conference on Machine Learning_ , pages 2961–2970, 2019.
  * Tampuu et al. [2017] A. Tampuu, T. Matiisen, D. Kodelja, I. Kuzovkin, K. Korjus, J. Aru, J. Aru, and R. Vicente.  Multiagent cooperation and competition with deep reinforcement learning.  _PloS one_ , 12(4):e0172395, 2017.
  * Gupta et al. [2017] J. K. Gupta, M. Egorov, and M. Kochenderfer.  Cooperative multi-agent control using deep reinforcement learning.  In _International Conference on Autonomous Agents and Multiagent Systems_ , pages 66–83. Springer, 2017.
  * Kermack and McKendrick [1927] W. O. Kermack and A. G. McKendrick.  A contribution to the mathematical theory of epidemics.  _Proceedings of the royal society of london. Series A, Containing papers of a mathematical and physical character_ , 115(772):700–721, 1927.
  * Li and Muldowney [1995] M. Y. Li and J. S. Muldowney.  Global stability for the seir model in epidemiology.  _Mathematical biosciences_ , 125(2):155–164, 1995\.
  * McCluskey [2010] C. C. McCluskey.  Complete global stability for an sir epidemic model with delay—distributed or discrete.  _Nonlinear Analysis: Real World Applications_ , 11(1):55–59, 2010.
  * Ogren and Martin [2000] P. Ogren and C. F. Martin.  Optimal vaccination strategies for the control of epidemics in highly mobile populations.  In _Proceedings of the 39th IEEE Conference on Decision and Control (Cat. No. 00CH37187)_ , volume 2, pages 1782–1787. IEEE, 2000.
  * Sattenspiel and Dietz [1995] L. Sattenspiel and K. Dietz.  A structured epidemic model incorporating geographic mobility among regions.  _Mathematical biosciences_ , 128(1-2):71–91, 1995\.
  * Lillicrap et al. [2015] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra.  Continuous control with deep reinforcement learning.  In _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2015.

##  Appendix A Environmental Modelling

###  A.1 Mobility

At each time step $t$, the mobility demands of all $n$ regions are represented
by a matrix $M^{t}_{\text{d}}\in\mathbb{R}^{n\times n}$, where
$M^{t}_{\text{d},i,j}$ denotes the mobility demand from region $i$ to region
$j$.

All actions at time step $t$ are arranged as a matrix $p^{t}\in[0,1]^{n\times
n}$, where $p^{t}_{i,j}$ denotes the proportion of mobility demand that can be
fulfilled from region $i$ to region $j$, which is decided by the destination
region $j$. Therefore, region $j$ is allowed to determine the $j$th column
$p_{j}\in[0,1]^{n}$ in matrix $p$.

Eventually, the actual allowed mobility $M^{t}_{\text{a}}$ at time step $t$ is
calculated by $M^{t}_{\text{a},i,j}=M^{t}_{\text{d},i,j}p^{t}_{i,j}$.

###  A.2 Pandemic Transmission

The pandemic transmission model is based on the Susceptible-Infected-
Hospitalized-Recovered (SIHR) model proposed in [8]. The pandemic state of
region $i$ at time step $t$ is
$E^{t}_{i}=(S^{t}_{i},I_{i}^{t},H_{i}^{t},R_{i}^{t})$, which consists of the
number of susceptible, infected, hospitalized, and recovered population within
that region, respectively. Agents are able to observe the number of
hospitalization and recovery, but cannot distinguish between susceptible
(healthy) people and infected people that are both asymptomatic. Therefore,
the visible pandemic state of region $i$ at time step $t$ is
$E^{t}_{\text{v},i}=(S^{t}_{i}+I_{i}^{t},H_{i}^{t},R_{i}^{t})$.

At each step $t$, pandemic transmission is calculated in two phases: mobility
happens first, and then the pandemic spreads within each region.

At the mobility happening stage, people move between regions according to
actual allowed mobility $M^{t}_{\text{a}}$, calculated in the previous
section. After the mobility happening stage, the intermediate state becomes
$\hat{E}^{t}_{i}=(\hat{S}^{t}_{i},\hat{I}^{t}_{i},\hat{R}_{i}^{t})$, which is
given by

$\displaystyle
E^{t}_{\text{s},i}=E^{t}_{i}-\sum_{j}\frac{M^{t}_{\text{a},i,j}}{N^{t}_{i}}E^{t}_{i},\quad
E^{t}_{\text{m},i}=\sum_{j}\frac{M^{t}_{\text{a},j,i}}{N^{t}_{j}}E^{t}_{j},\quad\hat{E}^{t}_{i}=E^{t}_{\text{s},i}+E^{t}_{\text{m},i}.$
(5)

Here $N_{i}^{t}$ is the population of region $i$ at time step $p$,
$E^{t}_{\text{m},i}$ stands for the pandemic state of the moving people, and
$E^{t}_{\text{s},i}$ stands for the pandemic state of the staying people.

At the pandemic spreading stage, the eventual pandemic state is calculated by

$\displaystyle
S^{t+1}_{i}=\hat{S}^{t}_{i}-\frac{\beta^{t}_{\text{s},i}S^{t}_{\text{s},i}I^{t}_{\text{s},i}}{N^{t}_{\text{s},i}}-\frac{\beta^{t}_{\text{m},i}S^{t}_{\text{m},i}I^{t}_{\text{m},i}}{N^{t}_{\text{m},i}},$
$\displaystyle
I^{t+1}_{i}=\hat{I}^{t}_{i}+\frac{\beta^{t}_{\text{s},i}S^{t}_{\text{s},i}I^{t}_{\text{s},i}}{N^{t}_{\text{s},i}}+\frac{\beta^{t}_{\text{m},i}S^{t}_{\text{m},i}I^{t}_{\text{m},i}}{N^{t}_{\text{m},i}}-\gamma\hat{I}^{t}_{i},$
(6) $\displaystyle
H^{t+1}_{i}=H^{t}_{i}+\gamma_{i}^{t}\hat{I}_{i}^{t}-\theta_{i}^{t}H_{i}^{t},$
$\displaystyle R_{i}^{t+1}=\hat{R}_{i}^{t}+\theta_{i}^{t}H_{i}^{t},$

where $\beta^{t}_{i}$ is the transmission rate of the susceptible people,
$\gamma^{t}_{i}$ is the hospitalization rate of the infected people, and
$\theta^{t}_{i}$ is the recovery rate of the hospitalized people.

###  A.3 Tolerance Levels

The local reward of an agent’s action on that region is a combination of the
pandemic-spread cost and the mobility-control cost. The pandemic-spread cost
$C_{\text{p},i}$ for agent $i$ is given by

$C_{\text{p},i}=k_{\text{h}}\exp\left(\frac{H_{i}}{H_{0,i}}\right),$ (7)

where $k_{\text{h}}$ is the hyperparameter that abstracts the cost induced by
the first observed unit of infection within the region, while $H_{0,i}$ is a
hyperparameter called pandemic-tolerance level that determines the
exponentially increasing rate of the cost as the number of hospitalized
patients increases. The mobility-control cost $C_{\text{m},i}$ of region $i$
is given by

$\displaystyle M^{t}_{\text{d},i}$
$\displaystyle=\sum_{j}M^{t}_{\text{d},j,i}\text{ },\quad
M^{t}_{\text{a},i}=\sum_{j}M^{t}_{\text{a},j,i}\text{ },$ $\displaystyle
L^{t}_{i}$
$\displaystyle=\sum_{\tau=0}^{t-1}\lambda^{t-\tau}\frac{M_{\text{d},i}^{t-\tau}-M_{\text{a},i}^{t-\tau}}{\bar{M_{\text{d},i}}},$
(8) $\displaystyle C^{t}_{\text{m},i}$
$\displaystyle=\exp\left(\frac{L_{i}^{t}}{L_{0,i}}\right)\frac{M_{\text{d},i}^{t-\tau}-M_{\text{a},i}^{t-\tau}}{\bar{M_{\text{d},i}}}.$

Here $M^{t}_{\text{d},i}$ is the total mobility demand into region $i$, while
$M^{t}_{\text{a},i}$ is the actual amount of mobility that is allowed to enter
region $i$ at time step $t$; $L^{t}_{0,i}$ is the cumulative penalty caused by
continuous mobility restriction of the same region, where $\lambda$ is the
temporal discount factor of historical penalties; $L_{0}$ is the
hyperparameter called lockdown-tolerance level that determines the
exponentially increasing rate of the cost as the allowed amount of mobility
decreases.

##  Appendix B Detailed Experimental Results

Detailed experimental results are displayed below in Table 2.

Table 2: Type-wise behavior patterns for agents of different types. (a) Type-
wise comprison table of mean action $\overline{p}$ for different mixing
ratios.

$\overline{\bm{p}}$ | $\bm{L_{+}}$ | $\bm{L_{-}}$
---|---|---
$\bm{H_{+}}$ | $1.000$ | $1.000$
$\bm{H_{-}}$ | $0.200$ | $0.801$
$\alpha=0.01$

$\overline{\bm{p}}$ | $\bm{L_{+}}$ | $\bm{L_{-}}$
---|---|---
$\bm{H_{+}}$ | $0.800$ | $1.000$
$\bm{H_{-}}$ | $0.261$ | $0.852$
$\alpha=0.40$

$\overline{\bm{p}}$ | $\bm{L_{+}}$ | $\bm{L_{-}}$
---|---|---
$\bm{H_{+}}$ | $0.720$ | $0.737$
$\bm{H_{-}}$ | $0.307$ | $0.850$
$\alpha=10.0$

(b) Type-wise comprison table of mean hospitalization $\overline{H}$ for
different mixing ratios.

$\overline{\bm{H}}$ | $\bm{L_{+}}$ | $\bm{L_{-}}$
---|---|---
$\bm{H_{+}}$ | $102.76$ | $103.84$
$\bm{H_{-}}$ | $0.00$ | $32.25$
$\alpha=0.01$

$\overline{\bm{H}}$ | $\bm{L_{+}}$ | $\bm{L_{-}}$
---|---|---
$\bm{H_{+}}$ | $89.00$ | $20.35$
$\bm{H_{-}}$ | $7.89$ | $7.93$
$\alpha=0.40$

$\overline{\bm{H}}$ | $\bm{L_{+}}$ | $\bm{L_{-}}$
---|---|---
$\bm{H_{+}}$ | $60.80$ | $8.54$
$\bm{H_{-}}$ | $31.22$ | $8.26$
$\alpha=10.0$