ifaamas [AAMAS ’23]Proc. of the 22nd International Conference on Autonomous
Agents and Multiagent Systems (AAMAS 2023)May 29 – June 2, 2023 London, United
KingdomA. Ricci, W. Yeoh, N. Agmon, B. An (eds.) 2023 2023 ???

# Artificial prediction markets present a novel opportunity for human-AI
collaboration

Anonymous submission

###### Abstract

Despite high-profile successes in the field of Artificial Intelligence,
machine-driven technologies still suffer important limitations, particularly
for complex tasks where creativity, planning, common sense, intuition, or
learning from limited data is required. These limitations motivate effective
methods for human-machine collaboration. Our work makes two primary
contributions. We thoroughly experiment with an artificial prediction market
model to understand the effects of market parameters on model performance for
benchmark classification tasks. We then demonstrate, through simulation, the
impact of exogenous agents in the market, where these exogenous agents
represent primitive human behaviors. This work lays the foundation for a novel
set of hybrid human-AI machine learning algorithms.

###### Key Words.:

prediction markets; machine learning; artificial intelligence; human-AI
collaboration

##  1 Introduction

A body of work on artificial prediction markets is emerging. These are
numerically simulated markets, populated by artificial agents (bot-traders)
for the purpose of supervised learning of probability estimators
barbu2012introduction. While nascent, this literature has demonstrated the
plausibility of using a trained market as a supervised learning algorithm,
achieving comparable performance to standard approaches on simple
classification tasks barbu2012introduction; barbu2013artificial;
jahedpari2014artificial; nakshatri2021design. In fact, these results are
sensible given the deep mathematical connections between prediction markets
and learning chen2008complexity; chen2010new; abernethy2011optimization.

Like other machine learning algorithms, functioning of an artificial
prediction market depends on several researcher-determined parameters: number
of agents; liquidity; initial cash; alongside parameters related to training
processes. Scenarios in which performance is robust or brittle to these
settings is yet unclear. Prior work has observed that artificial markets may
suffer from lack of participation rajtmajer2022synthetic. That is, like their
human counterparts in traditional prediction markets, agents may not invest in
the market if they do not have sufficient information Arrow:2008;
tetlock2008liquidity; rothschild2014extent; in practice, this occurs when an
asset representing test data point is too dissimilar to training examples.

In our view, the most promising opportunity afforded by artificial prediction
markets is eventual human-AI collaboration – a market framework should
theoretically support human traders participating alongside agents to evaluate
outcomes. Whether and how artificial prediction markets might benefit from
this hybrid scenario is an open question. The work we undertake here provides,
through simulation, initial support for this opportunity in the context of a
simple artificial market and primitive human behaviors.

Our work is framed by two primary research questions.

RQ1: How does performance of a simple artificial prediction market depend on
hyper-parameter selection?

RQ2: What impact does the inclusion of exogenous agents representing simple
(human-like) behaviors have on market performance?

Our findings support those of prior recent work indicating the promise of
artificial prediction markets for classification tasks. We demonstrate the
sensitivity of this approach to hyper-parameter selection and highlight, in
particular, the role of liquidity in moderating performance. Finally, we
demonstrate the exciting opportunity for _hybrid prediction markets_ to serve
as a framework for human-AI collaboration. We suggest that this approach may
be particularly valuable in contexts where machine learning falls short (e.g.,
lack of training data, complex tasks) and potential for human-only approaches
is either undesirable or infeasible.

##  2 Related Work

Our work builds upon and contributes to two primary literatures, namely, work
on artificial prediction markets and work in collaborative human-AI
technologies.

###  2.1 Artificial Prediction Markets

Prediction markets are simple futures markets used to aggregate disperse
information into efficient forecasts of uncertain future events
wolfers2004prediction; hanson2006information; manski2006interpreting;
wolfers2006interpreting. Specifically, market participants buy and sell
contracts that pay out based on the outcomes of future events. Market prices
generated from these contracts can be understood as a collective prediction
among market participants. Prediction markets have been successfully used,
e.g., for forecasting election outcomes berg2008prediction, sports betting
spann2009sports, forecasting infectious disease activity polgreen2007use, and
aggregating employee wisdom in corporate settings cowgill2009using;
gillen2012information.

Artificial prediction markets are a variation on this idea, wherein
numerically simulated markets populated by trained agents (bot-traders) are
used for the purpose of supervised learning of probability estimators
barbu2012introduction; barbu2013artificial. In initial formulations by Barbu
and Lay lay2010supervised; barbu2012introduction; lay2012artificial, each
agent is represented as a budget and a simple betting function. During
training, each agent’s budget is updated based on the accuracies of its
predictions over a training dataset. Authors found that these markets
outperformed standard approaches on benchmark classification and regression
tasks. Later, Storkey and colleagues storkey2011machine; storkey2012isoelastic
developed the so-called machine learning market, also for the purpose of
classification. In their formulation, each agent purchases contracts in order
to maximize a utility function. Most recently, Nakshatri et al.
nakshatri2021design proposed an artificial prediction market wherein agent
purchase logic is defined geometrically, in particular, by a convex semi-
algebraic set in feature space. Time varying asset prices affect the structure
of the semi-algebraic sets leading to time-varying agent purchase rules. Agent
parameters are trained using an evolutionary algorithm. Authors show that
their approach has desirable properties, e.g., the market satisfies certain
universal approximation properties, and there exist sufficient conditions for
convergence. Our work builds on this approach.

Like their human-populated counterparts, artificial prediction markets have
found a number of real-world applications barbu2013artificial;
jahedpari2014artificial. Ongoing theoretical work has offered support for
these promising experimental findings, highlighting the mathematical
connections between artificial markets and machine learning
chen2008complexity; ChenPennock:2010; abernethy2011optimization; hu2014multi.

###  2.2 Human-AI Collaboration

Despite high-profile successes in the field of Artificial intelligence (AI)
he2015delving; brown2019superhuman; kleinberg2018human; zhu2018human, machine-
driven solutions still suffer important limitations particularly for complex
tasks where creativity, common sense, intuition or learning from limited data
is required jarrahi2018artificial; lai2019human; green2019principles;
li2016crowdsourced; kamar2016directions; amershi2019guidelines;
muller2017organic. Both the promises and challenges of AI have motivated work
exploring frameworks for human-machine collaboration dellermann2021future;
wang2019human; nunes2015survey; puig2020watch; wu2022survey. The hope is that
we can eventually develop hybrid systems that bring together human intuition
and machine rationality to effectively and efficiently tackle today’s grand
challenges.

Recent work in hybrid intelligence systems has demonstrated the feasibility
and highlighted the potential of integrating human input into AI systems
kamar2016directions, or even, of human-AI collaboration wang2020human. The
spectrum of these efforts range from accounting for human factors in
technology design bansal2019beyond; canonico2019wisdom; harper2019role to
efficiently utilizing human inputs for training data amershi2014power in
applications as diverse as business nagar2011making; sowa2021cobots, civic
welfare fogliato2022case, criminal justice travaini2022machine, and healthcare
tschandl2020human; lee2021human; rajpurkar2022ai.

The work we describe here brings together the bodies of prior work on
artificial prediction markets and hybrid intelligence, proposing hybrid
prediction markets for direct integration of human wisdom into the deployment
of a machine learning algorithm.

##  3 Data

We consider three classification tasks. The first two are benchmark tasks used
broadly to compare performance of machine learning algorithms. The third is
the task of classifying scientific research outcomes as replicable or not
replicable – a challenging, complex task on which both machine learning
algorithms altmejd2019predicting; yang2020estimating; pawel2020probabilistic;
wu2021predicting and human assessment dreber2015using; camerer2016evaluating;
camerer2018evaluating; forsell2019predicting; gordon2020replication;
gordon2021predicting have achieved respectable but not excellent performance.
The replication prediction task, we suggest, is an example of the type of
problem well-suited to hybrid human-AI approaches and, specifically, the
hybrid prediction markets we propose.

###  3.1 Benchmark Machine Learning Datasets

The Iris dataset misc_iris_53 is one of the best known datasets in statistics
and machine learning, and one the earliest datasets used for evaluation of
classification methodologies. The dataset contains three classes of 50
instances each, where each class refers to a type of iris plant. One class is
linearly separable from the other; the latter are not linearly separable from
each other. Approaches based on support vector classification
mohan2020support, random forest classification mishina2015boosted;
chicho2021machine, and logisitc regression pinto2018iris have been reported to
work very well on this task, achieving $100\%$ or near-$100\%$ accuracy.

The Heart Disease dataset misc_heart_disease_45 is also a multivariate dataset
used for benchmark classification algorithms. Fourteen patient attributes are
used to predict presence or absence of heart disease. Random forest
singh2016heart, Xgboost rajadevi2021feature, and logistic regression
desai2019back achieve performance just under $90\%$ accuracy. While, support
vector classification achieves $86\%$ rajadevi2021feature.

###  3.2 Replication Studies Outcomes

In the last decade, several large-scale replication projects have been
undertaken across psychology, economics, political science, cancer biology and
other domains open2015estimating; camerer2016evaluating;
camerer2018evaluating; klein2014investigating; klein2018many;
cova2021estimating; errington2014open. Amongst their important impacts, these
studies have created small ground-truth datasets of replication studies
outcomes that can be used for train and test of automated approaches for
replication prediction. Specifically, we use the dataset and extracted
features considered by rajtmajer2022synthetic for ease of comparison. The
dataset containes 192 findings in the social and behavioral sciences, each
labeled either Replicable or Not Replicable, and a set of 41 features
extracted from each associated paper representing biblometric, venue-related,
author-related, statistical and semantic information. See wu2021predicting for
further detail on feature extraction processes.

Of note, authors in rajtmajer2022synthetic achieve $89.4\%$ accuracy,
remarkable for the task of replication prediction. However, accuracy is
calculated based on the approximately one-third of the test data that gets
evaluated by the market. Because agent participation is voluntary and agents
do not participate if they do not have sufficient information about a test
point, some (or much) of the data can be left unclassified. Our work uses the
same data and market structure described in rajtmajer2022synthetic. This
allows us to explore the effects of hyper-parameters (RQ1) and the inclusion
of exogenous agents (RQ2) on these performance/participation trade-offs.

##  4 Prediction Market Model

We use as a base model the artificial binary prediction market described in
nakshatri2021design. The state of the prediction market is defined by a pair
of integers $\mathbf{q}_{t}=(q^{0}_{t},q^{1}_{t})\in\mathbb{Z}_{+}^{2}$ giving
the number of units of the two asset classes that have been sold. For
simplicity we refer to the assets as $0$ and $1$. Traders are agents
$\mathcal{A}=\\{a_{1},\dots,a_{n}\\}$ who buy assets $0$ and $1$ using
policies $\\{\gamma_{1},\dots,\gamma_{n}\\}$. Also following
nakshatri2021design, we assume for simplicity that agents cannot sell. If
agent purchase policy $\gamma_{i}$ is conditioned on exogenous information
$\mathbf{x}\in D\subseteq\mathbb{R}^{n}$ then,
$\gamma_{i}:(\mathbf{q}_{t},\mathbf{x})\mapsto(r^{0},r^{1})$ and agent $i$
purchases $r^{0}$ units of $A_{0}$ and $r^{1}$ units of $A_{1}$, thus causing
a state update. In what follows, we assume that agents specialize in the
purchase of either Asset $0$ or Asset $1$ so that if $r^{0}>0$, then
$r^{1}=0$.

Asset prices are computed using a logarithmic market scoring rule (LMSR):

$\displaystyle p^{0}_{t}=\frac{\exp{(\beta q^{0}_{t})}}{\exp{(\beta
q^{0}_{t})}+\exp{(\beta q^{1}_{t})}}$ $\displaystyle
p^{1}_{t}=\frac{\exp{(\beta q^{1}_{t})}}{\exp{(\beta q^{0}_{t})}+\exp{(\beta
q^{1}_{t})}}.$

This is the softmax function of $(q^{0}_{t},q^{1}_{t})$. Liquidity $\beta$
adjusts the price change given a change in asset quantities
lekwijit2018optimizing. The fact that prices vary as a function of
$\mathbf{q}_{t}$ ensures that the policy need not take spot price into
consideration explicitly. It is often more convenient to work in units of
$1/\beta$ as $\beta$ can become arbitrarily close to zero. Experimental
results are therefore reported for this quantity as the liquidity factor.

To start the market, all agents may purchase assets at time $t=0$. After this,
we assume that agents arrive at the market with arrival rate $\lambda$ and
inter-arrival time governed by an exponential distribution. This allows us to
avoid scenarios in the hybrid setting where the synthetic traders swamp the
market.

The LMSR imposes a market maker price, so that actual trade costs are given
by:

$\displaystyle\kappa_{t}^{0}(\Delta
q^{0})=\frac{1}{\beta}\log\left\\{\frac{\exp[\beta(q^{0}_{t}+\Delta
q^{0})]+\exp[\beta q^{1}_{t}]}{\exp[\beta q^{0}_{t}]+\exp[\beta
q^{1}_{t}]}\right\\}$ $\displaystyle\kappa_{t}^{1}(\Delta
q^{1})=\frac{1}{\beta}\log\left\\{\frac{\exp[\beta
q^{0}_{t}]+\exp[\beta(q^{1}_{t}+\Delta q^{1})]}{\exp[\beta
q^{1}_{t}]+\exp[\beta q^{0}_{t}]}\right\\}.$

Here $\kappa^{i}_{t}(\Delta q^{i})$ is the cost to a trader for purchasing
$\Delta q^{i}$ units of Asset $i$ (with $i\in\\{0,1\\})$ at time $t$. For
small values of $\beta$ (large values of $1/\beta$) the cost of purchase
approaches the spot-price nakshatri2021design.

Agent purchase logic is governed by a time-varying bank value $B_{i}$ and a
characteristic function
$\psi_{i}:\mathbb{R}^{n}\times\mathbb{R}^{2}\times\mathbb{R}^{m}\to\mathbb{R}$
to reason about information $\mathbf{x}$ and its decision to buy an asset in
class $y_{i}$ is governed by:

$\Delta
q^{y_{i}}_{i}=H\left\\{\sigma[\psi_{i}(\mathbf{x},\mathbf{q};\bm{\theta})]-\kappa^{y_{i}}\right\\}\cdot
H\left(B_{i}-\kappa^{y_{i}}\right).$ (1)

Here $\sigma:\mathbb{R}\to[0,1]$ is a sigmoid function and $H(x)$ is the unit
step function defined as $0$ at $x=0$. The expression
$\sigma[\psi_{i}(\mathbf{x},\mathbf{q})]$ defines the value Agent $i$ places
on Asset $y_{i}$ as a function of the market state (and hence spot-prices) and
the information in the external information $\mathbf{x}$. If Agent $i$ places
more value on Asset $y_{i}$ than its present price $\kappa^{y_{i}}$, then
$H\left\\{\sigma[\psi_{i}(\mathbf{x},\mathbf{p})]-\kappa^{y_{i}}\right\\}=1$
and $\Delta q^{y_{i}}_{i}=1$ just in case the agent has sufficient funds given
by $H\left[B_{i}-\kappa^{y_{i}}\right]$. That is, Agent $i$ purchases a share
of Asset $i$. Notice we are assume that agents may buy one share of an asset
at a time. This both simplifies the agent logic and also would prevent the
agents from out-competing humans in the market in the hybrid scenario. The
vector $\bm{\theta}$ is a set of parameters that define the specific outputs
of $\psi_{i}$ and thus affect the agent purchase logic.

Let $\bm{\Theta}$ be the (matrix) of all parameter vectors for the agents.
After running for $T$ time units with input information $\mathbf{x}$, the spot
price for Asset $1$ is $p^{1}_{T}(\mathbf{x};\bm{\Theta})$. If we are given
input information $\\{\mathbf{x}_{1},\dots,\mathbf{x}_{N}\\}$ with class
information $\\{y_{1},\dots,y_{N}\\}$, then training the market is the process
of solving:

$\min_{\bm{\Theta}}\;\;\frac{1}{N}\sum_{j=1}^{N}\left\lvert
p^{1}_{T}(\mathbf{x}_{j};\bm{\Theta})-y_{j}\right\rVert^{2}.$

This problem is solved in nakshatri2021design using a genetic algorithm to
obtain a market that can classify external information $\mathbf{x}\in D$.

At the close of the market, the price of a each asset is taken as a proxy for
the market’s confidence in the corresponding outcome. In our binary market
model, there are two mutually exclusive possible outcomes and so the
(normalized) prices should sum to $1$. In this way, the market can be used for
regression or classification. In the three examples we consider here, the
market is used for classification. A separate market is run for each point in
the test set and the asset with the higher price is considered the market’s
classification decision for that test point.

We note, critically, that based on this model, agent participation is
voluntary and decision to participate is driven by $\Delta q^{y_{i}}_{i}=1$
from Equation 1. If this condition is not met during the course of the market
for any agents, there will be no market activity and thus no classification
decision for that test point. Authors in rajtmajer2022synthetic have noted
that this may occur frequently, particularly in cases where the training data
set is small or points in the test set are significant different from training
the data. Accordingly, we calculate accuracy and F1 based on the scored subset
of the data, while also reporting participation as a performance metric.

The artificial prediction market model includes five hyper-parameters that are
not optimized by the genetic algorithm discussed in nakshatri2021design:

  1. 1.

Agent inter-arrival rate ($\lambda$);

  2. 2.

Agent initial bank value ($B_{i}(0)$);

  3. 3.

Market liquidity ($1/\beta$);

  4. 4.

Simulation running time ($T$) or duration;

  5. 5.

Number of generations in the genetic (training) algorithm.

As such, these parameters are researcher-determined and warrant further study
(RQ1). Our first set of experiments, described below, explore the specific
roles of agent inter-arrival rate ($\lambda$), agent initial bank value
($B_{i}(0)$) (or, “cash”), and market liquidity ($1/\beta$) on performance. We
explore the robustness of performance to selection of these hyper-parameters,
highlighting accuracy and F1 score but also trade-offs with agent
participation.

In experiments that follow, the genetic algorithm is trained over five
generation. The objective function of the genetic algorithm maximizes root
mean square error of the estimated score. Agent performance is evaluated based
on profit; nonprofitable agents are deleted from the pool. The ten most
profitable agents are retained and, amongst them, the seven most profitable
agents are selected for mutation and crossover.

##  5 Experimental Design

The experiments we describe support the two primary research questions we have
laid out, respectively. First, we capture the effects of different
combinations of hyper-parameters on market performance (RQ1). Second, we
explore the impact of exogenous agents not trained through the evolutionary
training process, but rather who adopt one of a set of three simple purchasing
rules meant to represent primitive human inputs (RQ2).

###  5.1 Market robustness to hyper-parameters

We study the effects of inter-arrival rate $\lambda$, agent initial bank value
$B_{i}(0)$ (or, “cash”), and market liquidity $1/\beta$ on artificial market
performance. As mentioned, number of generations is fixed at five during
training; while, market duration is fixed at 20. These parameters were fixed
(vs. manipulated) to avoid combinatorial complexity during this initial study;
however, they should be further studied in future work. In practice, we have
found these values to be sufficient for market behavior to converge while also
offering reasonable run time. Liquidity is tested for the set of values
$\\{5,10,20,50,75,100,150,200,300\\}$. Initial cash is tested for
$\\{1,2,3,4,5,10,20\\}$; $\lambda$ is tested for
$\\{0.1,0.25,0.5,0.01,0.025,0.05,1\\}$. Our experiments consider all
combinations of these hyper-parameter values, $441$ total, and measure
corresponding performance in terms of accuracy, F1 score, and agent
participation (i.e., total number of scored test points). Performance for each
hyper-parameter set is determined based on 5-fold cross validation with 80/20
train/test splits. From these outcomes, we select best and worst-performing
hyper-parameter sets to be used for downstream analyses. This process is
outlined in Figure 10.

###  5.2 Market behavior with exogenous agents

We introduce three classes of exogenous agents representing simple,
fundamental behaviors which operate fully separate from the agent logic and
feature-based training protocol used for the other agents in the market. These
classes of behavior are intended to represent behavioral primitives that, in
combination, would underlie the actions of human participants in a hybrid
scenario. The first, _ground truth_ agents (GT) have perfect knowledge of the
correct outcome and always buy contracts corresponding to the correct outcome
whenever they have the opportunity to participate (which is moderated by their
arrival rate, $\lambda$). The second are _ground truth inverse_ agents
(GTinv). These agents also know the correct outcome but always buy contracts
corresponding to the incorrect outcome whenever they have an opportunity to
participate. This scenario is equivalent to the case where agents are simply
certain but incorrect in their forecast. Finally, our third class of agents
are _random_ agents which purchase contracts corresponding to one or the other
outcome randomly. Understanding that the decisions of human participants in
the hybrid prediction market would likely not exactly fall into any of these
three categories, we consider our simulations of markets with agents drawn
from these classes as drawing initial boundaries around the type of impact
human participants might have on the performance of an artificial market
depending on the complexity of the forecasting task, e.g., there may be a
classification task which is very easy for humans but difficult for algorithms
wherein we would expect near-perfect performance from human participants.

Our experiments measure impact of exogenous agents on market performance
measured, as before, by accuracy, F1 score, and agent participation. Because
exogenous agents are not trained, they are not subject to the genetic
algorithm. Rather, exogenous agents are added directly to the agent pool
during test. Our experiments test the impact of adding varying number of
agents from each class. We specify this number based on percentage of the
total agent pool. Specifically, we test hybrid market performance with the
inclusion of $\\{0.1\%,0.5\%,1\%\\}$ GT and GTinv agents. We test hybrid
market performance with the inclusion of random agents accounting for
$\\{1\%,5\%,10\%,50\%\\}$ of the total agent pool. Random agents are included
at a higher rate given the comparatively lesser impact they have on asset
prices.

##  6 Experimental Results

This section discusses the results obtained from all the experiments executed
on the three datasets mentioned above. We have analyzed the performance of the
artificial prediction market using different metrics to capture the degree of
coherence between observations and predictions. Owing to the size of these
datasets, we chose to go for 5-fold cross validation to ensure that we make
best use of the entire data and gather more accurate metrics in terms of the
performance of our markets across various hyper-parameter combinations. Our
market has been generalised to work like a binary classifier, therefore,
accuracy and F1 score were extracted as the performance evaluation metrics.
Accuracy is the fraction of correct predictions made by our model. For binary
classification, accuracy can also be calculated in terms of two positives
(True Positives(TP) and False Positives(FP)) and two negatives (True
Negatives(TN) and False Negatives(FN))as follows.

$\displaystyle Accuracy=\frac{TP+TN}{TP+TN+FP+FN}$

Apart from the accuracy of predictions, we also wanted to understand how well
our model classifies across the two classes. Hence, we chose F1-Score as our
evaluation metric since it combines the Precision and Recall of a classifier
(which means that it accounts for both FPs and FNs) into a single metric by
taking a harmonic mean as shown below.

$\displaystyle
F1-Score=2*\frac{\text{Precision}*\text{Recall}}{\text{Precision}+\text{Recall}}$

If a model produces no false positives then the precision of the model is 1.
F1-score ranges between 0 and 1. Higher the F1, better the model.

###  6.1 Analysis using IRIS Data

We started off our tests with the standard IRIS dataset which contains total
150 datapoints of 50 classes each as mentioned in Section 3.1. As shown in
Figure 10, we capture the average accuracy and average f1 for all
experimentally generated hyper-parameter combinations. Figure 1 highlights the
average F1 score with different combinations of $\lambda$, initial cash and
liquidity. From the colour datapoints in the figure, we can infer that the
best F1 scores are obtained when initial cash ranges from 1-4 and when
liquidity is greater than 100. We obtained a best case accuracy of 94.1% with
a F1 score 0.91 for the combination (liquidity: 300, lambda: 1, initial
cash:1). Hence, we a small subset was taken for deeper understanding to find
clearer patterns across combinations by fixing a pair from this set and
varying the third parameter.

From Table 1, 2, 3, we can clearly observe that initial cash and liquidity
tends to show a pattern in terms of the affect on performance of the market
while it’s relatively difficult to assess the impact of $\lambda$ on the
overall changes in captured metrics. The market performance increases as
liquidity increases (Table 3) but decreases when initial cash increases(Table
1). For example, the accuracy is 94% when the initial cash is 1 but it
decreases to 75% when the cash increased to 20.

Figure 2 showcases the average accuracy vs F1 score for the IRIS data with
different values of initial cash. In most of the cases, F1 scores are inline
with the accuracy and the best numbers are seen with an initial cash of 1
(highlighted with blue markers).

Table 1: Average F1 Score and Accuracy with different values of initial cash for best hyper-parameters of IRIS data Initial Cash | Liquidity | $\lambda$ | Accuracy | F1 Score
---|---|---|---|---
1.0 | 300.0 | 1.0 | 0.94 | 0.91
2.0 | 300.0 | 1.0 | 0.81 | 0.58
3.0 | 300.0 | 1.0 | 0.81 | 0.64
4.0 | 300.0 | 1.0 | 0.87 | 0.76
5.0 | 300.0 | 1.0 | 0.76 | 0.42
10.0 | 300.0 | 1.0 | 0.75 | 0.35
20.0 | 300.0 | 1.0 | 0.75 | 0.37
|  |  |  |
Table 2: Average F1 Score and Accuracy with different values of $\lambda$ for best hyper-parameters of IRIS data $\lambda$ | Liquidity | Initial Cash | Accuracy | F1 Score
---|---|---|---|---
0.100 | 300.0 | 1.0 | 0.87 | 0.79
0.250 | 300.0 | 1.0 | 0.87 | 0.80
0.500 | 300.0 | 1.0 | 0.87 | 0.78
0.010 | 300.0 | 1.0 | 0.87 | 0.79
0.025 | 300.0 | 1.0 | 0.91 | 0.86
0.050 | 300.0 | 1.0 | 0.88 | 0.82
1.000 | 300.0 | 1.0 | 0.94 | 0.91
|  |  |  |
Table 3: Average F1 Score and Accuracy with different values of Liquidity for best hyper-parameters of IRIS data Liquidity | $\lambda$ | Initial Cash | Accuracy | F1 Score
---|---|---|---|---
5.0 | 1.0 | 1.0 | 0.67 | 0.00
10.0 | 1.0 | 1.0 | 0.67 | 0.00
20.0 | 1.0 | 1.0 | 0.67 | 0.03
50.0 | 1.0 | 1.0 | 0.75 | 0.33
75.0 | 1.0 | 1.0 | 0.85 | 0.72
100.0 | 1.0 | 1.0 | 0.86 | 0.74
150.0 | 1.0 | 1.0 | 0.86 | 0.76
200.0 | 1.0 | 1.0 | 0.88 | 0.81
300.0 | 1.0 | 1.0 | 0.94 | 0.91
|  |  |  |
Figure 1: Average F1 with different values of $\lambda$, liquidity and initial
cash for Iris Data Figure 2: Average accuracy vs average F1 with different
values of Initial Cash for Iris Data

###  6.2 Analysis using Heart Disease Data

The complete set of 441 combinations were generated for heart disease dataset
as mentioned in Section 3.1. Figure 3 shows the average F1 score for all 441
combinations of hyper-parameters where the yellow colored datapoints represent
better F1 score than the magenda ones. These yellow colored datapoints are
spreaded all over the combinations of hyper-parameters set. Figure 4 describe
the obtained average accuracy with the average F1 score with different values
of $\lambda$. There is no effect of lambda on market performance as we can see
it is not possible to infer any pattern form the figure 4. The main intention
to plot the figure is to show there is no proper best/worse value of $\lambda$
regarding market performance. The best performance was obtained with accuracy
66% and F1 score 0.71 when only one datapoint was not scored by the model.
Now, to have more clear insight we took a small subset from the 441
combinations to explore more with these parameters.

The best market performance was obtained when the liquidity is in the range
(50-75), but there’s no defined pattern in the performance with change in
values for $\lambda$ and initial cash in case of Heart data. From table 4 we
can infer that with Only in case of increase in liquidity, effect the
performance of the model. For heart data when we have liquidity 50 the
obtained accuracy is 66% but it was reduced to 57% when liquidity is 200 which
means higher values of liquidity are not good for this dataset.

Table 4: Average F1 and Accuracy with different Liquidity for Heart Disease Data Liquidity | $\lambda$ | Initial cash | Accuracy | F1 score
---|---|---|---|---
50.0 | 0.05 | 20.0 | 0.66 | 0.71
20.0 | 0.05 | 20.0 | 0.60 | 0.66
75.0 | 0.05 | 20.0 | 0.61 | 0.64
200.0 | 0.05 | 20.0 | 0.57 | 0.62
100.0 | 0.05 | 20.0 | 0.58 | 0.62
300.0 | 0.05 | 20.0 | 0.59 | 0.61
150.0 | 0.05 | 20.0 | 0.58 | 0.59
10.0 | 0.05 | 20.0 | 0.54 | 0.56
5.0 | 0.05 | 20.0 | 0.59 | 0.54
|  |  |  |
Figure 3: Average F1 with different values of $\lambda$, liquidity and initial
cash for Heart Data Figure 4: Average accuracy vs average F1 with different
values of $\lambda$ for Heart Data

###  6.3 Analysis using Replication Data

We further repeated the phase 1 of the experiments on replication data to
understand how the market parameters change across datasets of varying
complexities and to capture similarities, dissimilarities and patterns across
datasets. This generalization would be helpful for future research in this
area. The average F1 score for all hyper-parameter combinations can be seen in
Figure 5. Figure 6 is the density plot for the replication data across the
accuracy and F1 score for all those 441 combinations.

Figure 8 shows the average accuracy with average F1 score for the replication
data with different values of initial cash. We observed that the market
performs well with initial cash of 1 and 20 respectively in some of the cases.
From these plots, we can further infer that there’s no clear best/worst
performing range of initial cash and lambda for the replication data as well.

Table 5: Agent participation with different values of Liquidity for Replication data Liquidity | Mean of agents participation
---|---
5 | 24.49
10 | 26.15
20 | 29.13
50 | 35.43
75 | 37.48
100 | 38.88
150 | 40.57
200 | 41.51
300 | 42.45
|
Table 6: Mean of Agent participation with different values of Initial Cash for Replication data Initial Cash | Mean of agents participation
---|---
1 | 35.39
2 | 35.87
3 | 35.22
4 | 34.96
5 | 34.79
10 | 34.90
20 | 34.73
|
Table 7: Mean of Agents participation with different values of $\lambda$ for Replication data $\lambda$ | Mean of agents participation
---|---
0.01 | 35.08
0.025 | 34.93
0.05 | 35.06
0.1 | 35.13
0.25 | 35.18
0.5 | 35.06
1 | 35.41
|
Table 8: First five best and worst Hyper-parameters for Replication Data Liquidity | $\lambda$ | Initial Cash | F1 | Accuracy | Scored%
---|---|---|---|---|---
5 | 0.05 | 1 | 0.84 | 0.79 | 36
10 | 0.05 | 10 | 0.84 | 0.76 | 35
5 | 1 | 4 | 0.83 | 0.75 | 37
5 | 0.1 | 1 | 0.83 | 0.77 | 36
5 | 0.1 | 2 | 0.83 | 0.75 | 37
150 | 0.25 | 4 | 0.69 | 0.58 | 35
100 | 0.05 | 2 | 0.68 | 0.0.57 | 35
150 | 0.5 | 20 | 0.68 | 0.58 | 35
10 | 0.5 | 3 | 0.65 | 0.66 | 55
75 | 0.1 | 2 | 0.64 | 0.65 | 52
|  |  |  |  |
Table 9: F1 score for different Exogenous Agents No-Ex | GT0.1% | GT0.5% | GT1% | GTinv0.1% | GTinv0.5% | GTinv1% | Random1% | Random5% | Random10% | Random50%
---|---|---|---|---|---|---|---|---|---|---
0.84 | 0.93 | 1 | 1 | 0.34 | 0.23 | 0.09 | 0.79 | 0.81 | 0.80 | 0.82
0.84 | 0.91 | 0.96 | 0.97 | 0.34 | 0.31 | 0.28 | 0.74 | 0.76 | 0.75 | 0.75
0.83 | 0.94 | 1 | 1 | 0.32 | 0.25 | 0.06 | 0.79 | 0.81 | 0.83 | 0.78
0.83 | 0.90 | 0.95 | 0.99 | 0.33 | 0.28 | 0.24 | 0.76 | 0.74 | 0.82 | 0.77
0.83 | 0.88 | 0.91 | 0.94 | 0.33 | 0.32 | 0.29 | 0.75 | 0.77 | 0.79 | 0.73
0.69 | 0.89 | 0.94 | 0.96 | 0.34 | 0.31 | 0.28 | 0.76 | 0.77 | 0.81 | 0.77
0.69 | 0.91 | 0.97 | 0.96 | 0.34 | 0.29 | 0.29 | 0.78 | 0.77 | 0.77 | 0.78
0.68 | 0.90 | 0.95 | 0.96 | 0.33 | 0.29 | 0.29 | 0.78 | 0.78 | 0.79 | 0.76
0.66 | 0.89 | 0.90 | 0.94 | 0.33 | 0.33 | 0.29 | 0.76 | 0.76 | 0.79 | 0.74
0.65 | 0.90 | 0.91 | 0.94 | 0.34 | 0.32 | 0.30 | 0.77 | 0.75 | 0.81 | 0.74
|  |  |  |  |  |  |  |  |  |
Table 10: Accuracy score for different Exogenous Agents No-Ex | GT0.1% | GT0.5% | GT1% | GTinv0.1% | GTinv0.5% | GTinv1% | Random1% | Random5% | Random10% | Random50%
---|---|---|---|---|---|---|---|---|---|---
0.79 | 0.92 | 1 | 1 | 0.27 | 0.14 | 0.05 | 0.76 | 0.77 | 0.77 | 0.78
0.76 | 0.90 | 0.95 | 0.97 | 0.27 | 0.21 | 0.17 | 0.67 | 0.68 | 0.67 | 0.70
0.75 | 0.94 | 1 | 1 | 0.26 | 0.14 | 0.03 | 0.76 | 0.77 | 0.80 | 0.77
0.77 | 0.89 | 0.95 | 0.99 | 0.26 | 0.18 | 0.14 | 0.70 | 0.66 | 0.77 | 0.73
0.75 | 0.87 | 0.90 | 0.94 | 0.27 | 0.23 | 0.18 | 0.67 | 0.70 | 0.74 | 0.68
0.58 | 0.88 | 0.93 | 0.95 | 0.26 | 0.21 | 0.17 | 0.70 | 0.70 | 0.76 | 0.71
0.57 | 0.90 | 0.96 | 0.95 | 0.25 | 0.18 | 0.18 | 0.73 | 0.70 | 0.71 | 0.72
0.58 | 0.89 | 0.94 | 0.95 | 0.25 | 0.19 | 0.17 | 0.73 | 0.71 | 0.73 | 0.74
0.65 | 0.88 | 0.89 | 0.93 | 0.25 | 0.26 | 0.18 | 0.71 | 0.68 | 0.74 | 0.69
0.64 | 0.89 | 0.90 | 0.94 | 0.27 | 0.23 | 0.19 | 0.72 | 0.68 | 0.76 | 0.70
|  |  |  |  |  |  |  |  |  |
Figure 5: Average F1 with different values of Lambda, liquidity and initial cash for Replication Data Figure 6: Density plot for replication data across Accuracy and F1 score Figure 7: Average accuracy vs average F1 with different values of Liquidity for Replication Data Figure 8: Average accuracy vs average F1 with different values of Initial Cash for Replication Data Figure 9: F1 with different percentage of added GT inverse agents for replication data Figure 10: Phase 1 Architecture Figure 11: Phase 2 Architecture Figure 12: F1 with different percentage of added Random agents for replication data Figure 13: F1 with different percentage of added GT agents for replication data Table 11: Agent participation with different percentages of GT agents No-Ex | GT0.1% | GT0.5% | GT1%
---|---|---|---
4112 | 4543 | 4671 | 5352
3280 | 3766 | 4266 | 5339
3431 | 3325 | 4057 | 5132
4142 | 4442 | 5244 | 5496
3581 | 3989 | 4638 | 5023
5963 | 6462 | 6934 | 7544
5802 | 5784 | 6356 | 7028
6076 | 6100 | 6790 | 7475
3906 | 4063 | 4286 | 5180
6448 | 5436 | 5873 | 6766
|  |  |

The most impactful hyper-parameter we found after doing the complete study is
the liquidity for all three datasets. Figure 7 highlights the average accuracy
vs average F1 score with different values of liquidity for the replication
data. For all these three datasets the observation is same and it is evident
that the performance of the market doesn’t depend on the lambda. The best
performance was observed with accuracy 79% and F1 score 0.84 when the initial
cash was 1 and liquidity was 5. The total number of papers were 145 out of
which 93 were not scored and 52 were scored.

Figure 14: Agent Participation with different percentage of added GT agents for replication data Table 12: Agent participation with different percentages of GTinv agents No-Ex | GTinv0.1% | GTinv0.5% | GTinv1%
---|---|---|---
4112 | 4284 | 5374 | 6785
3280 | 3636 | 4512 | 5484
3431 | 3928 | 4664 | 5299
4142 | 4085 | 5169 | 7010
3581 | 4067 | 4883 | 6409
5963 | 6207 | 6414 | 7803
5802 | 5985 | 6174 | 7448
6076 | 6218 | 6529 | 7430
3906 | 4045 | 5042 | 6048
6448 | 5607 | 6120 | 7228
|  |  |
Figure 15: Agent Participation with different percentage of added GT inverse
agents for replication data

Another very interesting observation is the mean of agent participation with
different values of liquidity which is mentioned in Table 5. The agent
participation clearly increases with increase in the value of liquidity.
Liquidity constant directly impacts the shifts in the price.The agents tend to
participate more as the price shifts tends to increase. Hence, this behavior
is inline with our expectations. We captured similar data for different values
of lambda and Initial cash as shown in Table 7 and Table 6 respectively. The
mean of the agent participation tends to show no significant variation with
change in lambda and initial cash and hence this supports our previous
observation that these parameters don’t produce a consistent decipherable
effect on the performance of the market.

At the end of our experiments, with the hyper-parameter combinations we
understood that there is no fixed best hyper-parameter set for all kinds of
datasets. Every researcher needs to experiment and set the best hyper-
parameter combination for their application to get the best performance from
the prediction market. For the next set of experiments, we decided to take the
best and worst 5 combinations of hyper-parameters for the replication dataset
which is listed in Table 8. The prediction market classifies a subset of the
input papers with good accuracy and F1 score. The last column shows the total
percentage of papers scored for each of these combinations. The highest number
of scored papers is seen with a (liquidity, $\lambda$, initial cash)
combination of (10, 0.5, 3), when the market was able to get 55% scored papers
with 65% accuracy and with an F1 of 0.66.

####  6.3.1 Discussion RQ1

Bring together.

###  6.4 Analysis using Exogenous Agents

In the second phase of the study, we have introduced simulated exogenous
agents to analyse their effect on the market performance. For this purpose, we
chose to inject incremental percentages of simulated agents to track the
shifts in accuracy, F1-score and eventually the overall agent participation.
Our agents are divided into three categories - Ground truth (GT), Ground Truth
Inverse(GT Inv) and Random agents respectively. The accuracy and F1-score for
the top 5 best and worst combinations after the introduction of these agents
has been listed down in Table 10 and Table 9 respectively.

####  6.4.1 Introduction of Ground Truth Agents in the Agent Pool

Agents with Ground Truth information have the potential to causes sudden
changes on how the market behaves. Hence, we selected very small percentages
of such agents in our tests.

$\%\>of\>GT\>agents=\\{0.1,0.5,1\\}$

In the set above, 0.1% implies that we would inject 1 GT agent for every 1000
agents currently in the pool.

From Table 9 and Table 10, it is evident that even a small amount of GT agents
can improve the market performance substantially. Figure 13 clearly charts
down this incremental improvement. On injecting 0.1% of GT agents, we see a
spike in F1 score from 0.84 to 0.93 and the accuracy shoots from 79% to 92% as
shown in the tables above. Further gradual addition of such agents ensures
100% accuracy and F1 for the best combinations. The impact of such agents
becomes more evident in the cases where the previously worst performing set of
hyper-parameters now showed an improvement in accuracy from 64% to 89% with
just 0.1% of GT agents.

####  6.4.2 Introduction of Ground Truth Inverse Agents in the Agent Pool

Similar to GT agents, GTinv agents can intuitively cause sudden drops in
performance. These drops were clearly seen in our tests and can be tracked
from Table 9 and Table 10 respectively. Just like GT Agents, we chose the
following set of incremental percentages for GTinv agents since they drive the
market in a defined direction.

$\%\>of\>GTInv\>Agents=\\{0.1,0.5,1\\}$

Figure 9 plots the F1 score for different percentage of GTinv agents. The
black bar represents the case where there are no GTinv agents in the market.
It is clearly visible from the bar plot that market performance deteriorates
with the increase in amount of GTinv agents. Addition of just 0.1% of GTinv
agents dropped the accuracy from 79% to 27%. The accuracy further drops down
to 14% after adding 0.5% of agents and any further addition pushes the
accuracy down to zero as per our tests. Similar behavior is seen in the F1
scores as well. The best F1 score is 0.84 prior to the introduction of GTinv
agents but it drops down to 0.34 with just 0.1% of such agents.

####  6.4.3 Random Agents

We further created a set of simulated agents which weren’t trained on the
default feature set to simulate real-time human experiments. We found that
it’s infeasible to assess a pattern in the market performance owing to the
randomness introduced by these agents. Figure 12 shows how the F1 score
changes when these Random agents are injected to the training pool. We have
used a different set of percentages for these agents as shown below.

$\%\>of\>Random\>Agents=\\{1,5,10,50\\}$

As expected, the F1 scores don’t showcase a clear shift in any direction as
can be seen from Figure 12, but these agents have the potential to cause an
overall change in the market performance. The impact of these agents varies
across best to worst choices of markets. Adding 10% Random agents drops the
F1-score from 0.84 to 0.80 for the best case but it also improves the score
from 0.65 to 0.81 for the previously worst performing combination. Similar
behavior is seen if we add 5% random agents as it changes F1-score from 0.84
to 0.81 for the best case but it improves from 0.65 to 0.75 for the worst case
scenario. On increasing the percentage to 50%, we see a further dip in
performance compared to the case with just 10% random agents But if you
increase it to 50% then the overall performance will go down from the results
obtained for 10% random agents.

####  6.4.4 Agent Participation varies with added Exogenous agents

The addition of these agents provides an opportunity to derive various
intuitions. One such analysis was how the agent participation changes with
added GT and GTinv agents. Agent participation increases with the increase in
percentage of these agents in the market as they have the potential to drive
other agents by bidding in defined directions. Table 11 and Table 12 shows
total agent participation for different percentages of GT agents and GTinv
agents respectively. From Figure 14, we can observe that participation was
highest for 1% of newly added GT agents and lowest for no exogenous agents for
all 10 combinations. If we take the first combination for example, when there
are no exogenous agents the participation was 4112, for 0.1% GT agents
participation increases to 4543 and then it became 5352 for 1% added GT agents
during market run. Similarly, Figure 15 shows total agent participation for
different percentage of added GTinv agents. It can be seen that incremental
addition of GTinv agents drives increased agent participation for all 10
combinations. For example, if we chose the last set of hyper-parameter
combination the agent participation was 6448 for no exogenous agents but it
increased to 7228 on adding 1% of GTinv agents.

####  6.4.5 Discussion RQ2

##  7 Expected Contributions

This study provides the research community with a new approach to the
prediction market that has not been explored yet. The analysis of the
simulated humans with trained AI agents and how they behave together in the
field of research reproducibility is new. We have tried to analyze the
scenario when there will be real human participants in the prediction market
and will buy/sell alongside the trained AI agents and how their participation
will affect the market performance as well as the agent participation. The
comparative study of the artificial prediction market and the market with
exogenous agents (also known as simulated humans) point out the need for
human-AI collaboration in the prediction market which should motivate future
researchers to explore more in the field of Human-AI study. The overall study
with the hyper-parameters will be very helpful for future research because it
will save time and effort. They don’t have to run multiple experiments to
understand the behavior of these parameters. They will have a clear idea about
the experiments needed for their dataset to choose the best set.

The proposed research could have an impact on research society in broad ways
because the prediction market has been massively used in almost every research
domain. Scientific research reproducibility is one of the most sensitive areas
for all researchers. The proposed research will produce a clear vision of the
future hybrid prediction market. A lot of money and time will be saved.
Previously it was observed that there was a lot of loss in the market due to
replication failures. People can use that money and time for other productive
research which can be used in different ways for the betterment of society.

##  8 Conclusions

Previous research suggested that artificial prediction markets work wonders in
various applications. In many cases, they perform better than human
predictions. But some of the applications are very complex and they need human
intelligence for better understanding. In these situations, the combined
prediction of humans and AI agents can work wonders as per our hypothesis.
Therefore, we have experimented with simulated humans in the artificial
prediction market to explore how the participation of different exogenous
agents will affect the market performance. The artificial prediction market
with random agents, GT agents, and GT inverse agents was analyzed to
understand the bounds of the behavior of the market in a hybrid setting. After
adding a different percentage of exogenous agents, the performance of both
markets (artificial prediction market and market with simulated humans) was
compared using different evaluation matrices. Trained market agents will not
participate if they don’t have enough similar training features. In those
cases, human participation can drive agent participation in the market. The
final hope of this research study is that it will encourage other researchers
to explore more in Human-AI collaboration more. Also, we provided an important
understanding of different hyper-parameters and how they differ for different
datasets. Future researchers will have a complete understanding of how to use
these parameters to improve the performance of the prediction market. For
future work, we will introduce human participants in our prediction market
with AI agents to investigate the real scenario.