# Leveraging automatic strategy discoveryto teach people how to select better
projects

Lovis Heindrich1∗, Falk Lieder1,2

( 1Max Planck Institute for Intelligent Systems, Tübingen, Germany
2Department of Psychology, UCLA, Los Angeles, USA
∗E-mail<EMAIL_ADDRESS>
)

###### Abstract

The decisions of individuals and organizations are often suboptimal because
normative decision strategies are too demanding in the real world. Recent work
suggests that some errors can be prevented by leveraging artificial
intelligence to discover and teach prescriptive decision strategies that take
people’s constraints into account. So far, this line of research has been
limited to simplified decision problems. This article is the first to extend
this approach to a real-world decision problem, namely project selection. We
develop a computational method (MGPS) that automatically discovers project
selection strategies that are optimized for real people and develop an
intelligent tutor that teaches the discovered strategies. We evaluated MGPS on
a computational benchmark and tested the intelligent tutor in a training
experiment with two control conditions. MGPS outperformed a state-of-the-art
method and was more computationally efficient. Moreover, the intelligent tutor
significantly improved people’s decision strategies. Our results indicate that
our method can improve human decision-making in naturalistic settings similar
to real-world project selection, a first step towards applying strategy
discovery to the real world.

##  1 Introduction

Corporations and individuals commonly have to select a project out of multiple
alternatives. These project selection problems are usually high-stakes
decisions that can be highly impactful for the future of an organization. For
example, an organization looking for a sustainable investment project
(Khalili-Damghani and Sadi-Nezhad, 2013) can benefit both financially and by
improving its public image by selecting an impactful and profitable project,
or incur major losses by selecting an unsuitable project.

Previous research on project selection recommends that candidate projects
should be evaluated by multiple experts (Coldrick et al., 2002, Khalili-
Damghani and Sadi-Nezhad, 2013, Liu et al., 2017), and many structured
approaches to integrate the experts’ opinions exist (de Souza et al., 2021).
However, existing project selection techniques are not well utilized in the
real world (Schmidt and Freeland, 1992, Liu et al., 2017, Henriksen and
Traynor, 1999). We hypothesize this underutilization is partly caused by the
complexity and implementation costs of normative project selection strategies.
Scoring approaches, which score alternative projects on a number of relevant
selection criteria, have been proposed as a less complex selection method that
can be applied to real-world problems more easily (Henriksen and Traynor,
1999, Khalili-Damghani and Sadi-Nezhad, 2013, Coldrick et al., 2002, Sadi-
Nezhad, 2017). However, even scoring approaches face significant challenges
when applied to real-world problems: precisely evaluating projects on multiple
criteria can be a time-consuming and expensive process, but existing selection
techniques often pay little attention to the information-gathering costs and
instead assume expert opinions on all criteria are readily available and
instead focus on how to integrate existing expert evaluations into a final
project choice (Abdel-Basset et al., 2019, Khalili-Damghani and Sadi-Nezhad,
2013).

Due to the limited utilization of complex project selection methods, decision-
makers, therefore, often rely on their intuition and much simpler techniques
like brainstorming (Kornfeld and Kara, 2013). This is concerning because the
intuitive decisions of groups and individuals are highly susceptible to biases
and unsystematic errors (Kahneman et al., 1982), such as weighing one’s own
opinion too strongly (Yaniv and Kleinberger, 2000), being overconfident, and
weighting the recommendations of advisors by their confidence rather than
their competence (Bonaccio and Dalal, 2006). However, people’s errors in
decision-making can partly be prevented by teaching prescriptive decision
strategies adapted to humans by taking their cognitive limitations into
account (Lieder and Griffiths, 2020, Callaway et al., 2022). This approach to
improving human decision-making is known as boosting (Hertwig and Grüne-
Yanoff, 2017).

To identify appropriate decision strategies, we can score candidate strategies
by their resource-rationality, that is, the degree to which they make rational
use of people’s limited time and bounded cognitive resources (Lieder and
Griffiths, 2020). In the resource-rational framework, the decision operations
people can perform to arrive at a decision are modeled explicitly and assigned
a cost. The overall efficiency of a decision strategy $h$ in an environment
$e$ can then be computed by subtracting the expected costs $\lambda$ of the
$N$ used decision operations from the expected utility $R_{total}$ of the
resulting decision (see Equation 1) (Consul et al., 2022). This measure is
called resource-rationality score (RR-score) (Consul et al., 2022). People are
usually not fully resource-rational, and identifying decision strategies would
enable people to perform as well as possible is an important open problem
(Callaway et al., 2022, Consul et al., 2022, Heindrich et al., 2023, Mehta et
al., 2022).

$\text{RR}(h,e)=\mathbb{E}[R_{total}|h,e]-\lambda\mathbb{E}[N|h,e]$ (1)

Recent work has demonstrated that the theory of resource rationality makes it
possible to leverage AI to automatically discover and teach decision
strategies that enable real people to make their decisions as well as possible
(Callaway et al., 2022, Consul et al., 2022, Becker et al., 2022, Skirzyński
et al., 2021, Mehta et al., 2022). This approach has been dubbed AI-powered
boosting. The first step of AI-powered boosting is to compute resource-
rational decision strategies. Automatic strategy discovery methods (Callaway
et al., 2022, Consul et al., 2022, Heindrich et al., 2023, Skirzyński et al.,
2021, Mehta et al., 2022) can discover efficient decision strategies by
solving the metareasoning problem of deciding which decision operations to
perform. While recent work has extended automatic strategy discovery methods
to larger (Consul et al., 2022) and partially observable environments
(Heindrich et al., 2023), so far, they have not been applied to real-world
decisions such as project selection.

Developing computational methods for decision support is one of the core
problems of operations research (Gupta et al., 2022, Eom and Kim, 2006). In
this article, we present a novel decision support method with the goal of
discovering prescriptive decision-strategies that can be taught to people to
improve how they select projects. Project selection is challenging because
many crucial attributes of the candidate projects, such as their expected
profitability, cannot be observed directly. Instead, they have to be inferred
from observations that are not fully reliable. We, therefore, formalize
project selection strategies as policies for solving a particular class of
partially observable Markov Decision Processes (POMDPs). This formulation
allows us to develop the first algorithm for discovering resource-rational
strategies for human project selection. To achieve this, we model a realistic
project selection task as a metareasoning problem. The metareasoning consists
in deciding which information one should request from which advisors when
information is highly limited, uncertain, and costly. We develop an efficient
algorithm for solving this problem based on MGPO, an algorithm for discovering
resource-rational strategies for human decision-making in partially observable
environments (Heindrich et al., 2023), and apply it to derive a prescriptive
decision strategy for a project selection problem a financial institution
faced in the real world (Khalili-Damghani and Sadi-Nezhad, 2013). Finally, we
develop an intelligent tutor (Callaway et al., 2022) that teaches the decision
strategy discovered by our method to people. We evaluated our approach by
letting our intelligent tutor teach the automatically discovered project
selection strategy to about 100 people, and then evaluated the quality of
their decisions in realistic project selection problems against two control
groups. Our results indicate that our approach can successfully improve human
decisions in real-world problems where people are reluctant to let machines
decide for them.

##  2 Background

#### Project selection

In the project selection problem, a decision-maker aims to select the best-
fitting project out of several candidates (Sadi-Nezhad, 2017). Apart from a
project’s profitability, the evaluation usually also considers other factors,
such as the alignment with organizational goals (Carazo et al., 2012). This
problem can be formalized as multi-criteria decision-making (MCDM) (de Souza
et al., 2021, Mohagheghi et al., 2019). Projects can be evaluated using a
scoring technique, which evaluates relevant criteria and then combines them to
a weighted sum (Sadi-Nezhad, 2017). Common approaches to solving the project
selection problem include techniques such as the analytic hierarchy process,
the analytic network process, real options analysis, and TOPSIS (see (de Souza
et al., 2021) for a review). These methods are commonly combined with fuzzy
sets to account for uncertainty (Khalili-Damghani and Sadi-Nezhad, 2013).
However, these methods are rarely used in real-world problems because
implementing them would require gathering and integrating a lot of information
through a time-consuming process, which is often incompatible with the
organizational decision process (Schmidt and Freeland, 1992, Liu et al.,
2017). Instead, organizations often rely on simpler, less structured methods
like brainstorming (Kornfeld and Kara, 2013). In addition, while the detailed
information required by these methods can be costly and time-consuming to
acquire in real-world settings, the methods often don’t take into account how
much information is needed to make an efficient decision.

#### Judge-advisor systems

In a Judge Advisor System (JAS) (Bonaccio and Dalal, 2006), typically, a
single decision-maker has to make a decision, and multiple advisors support
the decision-maker by offering advice. Variations of the task can include
costly advice (Yaniv and Kleinberger, 2000, Gino, 2008), or advisors with
varying reliability (Olsen et al., 2019). This is a common situation when CEOs
decide which project their company should launch next. Unfortunately,
decision-makers are known to be highly susceptible to systematic errors, such
as weighing one’s own opinion too strongly, overconfidence, egocentric advice
discounting, and weighting the recommendations of advisors by their confidence
rather than their competence (Bonaccio and Dalal, 2006, Ronayne et al., 2019,
Yaniv and Kleinberger, 2000). We model the project selection problem within
the JAS framework by letting project evaluators take the role of advisors with
varying reliability.

#### Resource rationality

Resource-rational analysis is a cognitive modeling paradigm used to describe
optimal decision-making with limited computational resources (Lieder and
Griffiths, 2020). The paradigm considers not only the quality of decisions,
but also the computational costs of reaching them. According to this paradigm,
good decision-making consists in making efficient use of the available
computational resources by balancing the competing objectives of maximizing
the expected decision quality and minimizing computational costs. Resource
rationality can be used to describe the quality of a decision strategy by
subtracting the cost of the cognitive operations from the expected reward of
the resulting decision. This measure is called resource-rationality score (RR-
score) (Consul et al., 2022). It will be used throughout this article to
evaluate and compare alternative decision strategies. People are usually not
fully resource-rational, and figuring out what decision strategies would
enable people to perform as well as possible is an important open problem
(Callaway et al., 2022, Consul et al., 2022, Heindrich et al., 2023, Mehta et
al., 2022).

#### Strategy discovery methods

Discovering resource-rational planning strategies can be achieved by solving a
meta-level Markov decision process (Hay et al., 2014, Callaway et al., 2018a,
b), which models the metareasoning process as a Markov Decision Process (MDP),
which state represents the current belief about the environment and actions
represent decision operations. Performing a decision operation results in a
negative cost and results in an update to the belief state. A special
termination action represents exiting the metareasoning process and making a
real-world decision, guided by the current beliefs (Hay et al., 2014).
Multiple methods for solving meta-level MDPs exist (e.g. (Russell and Wefald,
1991, Callaway et al., 2018a, Hay et al., 2014, Consul et al., 2022)). We
refer to these algorithms as strategy discovery methods (Callaway et al.,
2018a, 2022, Skirzyński et al., 2021, Consul et al., 2022, Heindrich et al.,
2023, Mehta et al., 2022). They learn or compute policies for selecting
sequences of cognitive operations (i.e., computations) people can perform to
reach good decisions.

MGPO (Heindrich et al., 2023) is currently the only strategy discovery
algorithm that can efficiently approximate resource-rational strategies for
decision-making in partially observable environments. MGPO chooses decision
operations by approximating their value of computation (Russell and Wefald,
1991) in a myopic manner: it always selects the computation whose execution
would yield the highest expected gain in reward if a decision had to be made
immediately afterward, without any additional planning. MGPO makes use of the
myopic approximation due to the high computational complexity of partially
observable Markov Decision Processes (POMDP). Alternative approaches to
modeling information gathering POMDPs are difficult to apply to partially
observable meta-level MDPs, as they require carefully designed information
rewards (Boutilier, 2002) or simplify the problem by restricting environments
to be symmetric (Doshi and Roy, 2008).

#### Cognitive tutors

Past work has developed cognitive tutors that teach automatically discovered
planning strategies to people (Callaway et al., 2022, Skirzyński et al., 2021,
Mehta et al., 2022, Consul et al., 2022). Training experiments indicated that
training with these cognitive tutors could significantly boost the quality of
people’s planning and decision-making (Callaway et al., 2022, Skirzyński et
al., 2021, Consul et al., 2022, Heindrich et al., 2023, Mehta et al., 2022).
These cognitive tutors teach efficient decision strategies in an automated
manner, usually by computing the value of available decision operations using
strategy discovery methods, and providing the learner feedback on the quality
of the computations they select. Initially limited to small planning tasks due
to the computational complexity of solving meta-level MDPs (Lieder et al.,
2019, Callaway et al., 2022), recent work has extended existing methods to
larger (Consul et al., 2022) and partially observable (Heindrich et al., 2023)
settings. However, none of these methods have been applied to naturalistic
problems so far.

A crucial obstacle is that these methods are limited to settings where all
decision-relevant information comes from the same source. By contrast, in the
real world, people have to choose between and integrate multiple different
sources of information. In doing so, they have to take into account that some
information sources are more reliable than others. Additionally, current
strategy discovery methods are limited to artificial settings where each piece
of information is an estimate of a potential future reward. By contrast, in
the real world, most information is only indirectly related to future rewards,
and different pieces of information have different units (e.g., temperature
vs. travel time).

##  3 Formalizing optimal decision strategies for human project selection as
the solution to a meta-level MDP

In this section, we introduce explain our general resource-rational model of
project selection, which we expect to be widely applicable to concrete, real-
world project selection problems.

Our model of project selection consists of two decision problems, an object-
level decision-problem and a meta-level MDP (Callaway et al., 2018a, Hay et
al., 2014). The two decision problems separate the actions the decision-maker
has to choose between (object level), such as executing one project versus
another, from decision operations that represent thinking about which of those
object-level actions to perform (meta-level), such as gathering information
about the projects’ attributes. This allows us to solve both problems
separately. The object-level decision problem is a MCDM problem, where a set
of $N_{P}$ potential projects $\mathcal{P}=\\{p_{1},...p_{N_{P}}\\}$ are
evaluated using $N_{C}$ relevant criteria $C=[c_{1},...c_{N_{C}}]$ weighted by
fixed predetermined weights $W=[w_{1},...w_{N_{C}}]$. Actions in the object-
level problem represent selecting the corresponding project
($\mathcal{A}=\\{a_{1},...,a_{N_{P}}\\}$). The reward of a selecting a project
is computed by summing the weighted criteria scores of the selected project:
$r^{O}(a_{i})=\sum_{c}w_{c}c_{c,i}$ (Coldrick et al., 2002).

While the object-level decision problem is our model of the project selection
task, the meta-level MDP is our formalization of the problem of discovering
resource-rational project selection strategies. It models the task of
gathering information about deciding which project to select. States in the
meta-level MDP are belief states that represent the current information about
each project’s attributes. We model belief states using a multivariate Normal
distribution to quantify the estimated value and uncertainty about the $N_{P}$
projects’ scores on the $N_{C}$ criteria:
$b=\left[\left(\mu_{1,1},\sigma_{1,1}\right),\cdots,\left(\mu_{N_{P},N_{C}},\sigma_{N_{P},N_{C}}\right)\right]$.
The actions (decision operations) of the meta-level MDP gather information
about the different attributes of projects by asking one of the $N_{E}$
experts for their estimate of how one of the projects scores on one of the
criteria. Experts provide discrete estimates from $min_{obs}$ to $max_{obs}$,
and expert estimates can differ in their reliability and their cost.
Specifically, the available actions are
$A^{M}=\left\\{a_{1,1,1},\cdots,a_{N_{P},N_{C},N_{E}},\perp\right\\}$, where
the meta-level action $a_{i,j,k}$ represents asking the expert $e_{k}$ for
their estimate of criterion $c_{j}$ of project $p_{i}$. After receiving
information $obs$ from an expert, the current belief state
$b_{p_{i}c_{j}}=\mathcal{N}(\mu,\sigma)$ is updated by applying the update
equation for a Gaussian likelihood function with standard deviation
$\sigma_{e}$ (i.e. the expert’s reliability) and a conjugate Gaussian prior
(i.e., the current belief), that is
$\hat{\mu}\leftarrow\left(\frac{w_{c_{i}}\cdot\mu}{(w_{c_{i}}\cdot\sigma)^{2}}+\frac{w_{c_{i}}\cdot
obs}{(w_{c_{i}}\cdot\sigma_{e})^{2}}\right)\cdot\left((w_{c_{i}}\cdot\sigma)^{2}+(w_{c_{i}}\cdot\sigma_{e})^{2}\right)$
and $\hat{\sigma}\leftarrow\sqrt{\frac{1}{\frac{1}{(\sigma\cdot
s)^{2}}+\frac{1}{(\sigma_{e}\cdot s)^{2}}}}$.

The reward of these meta-level actions is the negative cost
$r^{M}(a_{i,j,k})=-\lambda_{e_{k}}$ of asking the expert $e_{k}$ for help.
Finally, the meta-level action $\perp$ is the termination action, which
corresponds to terminating the decision-making process and selecting a
project. The reward of the termination action is the expected reward of
selecting the best project according to the current belief-state. An optional
budget parameter $N_{a}$ specifies the maximum number of available meta-level
actions, after which the termination action is performed automatically.

Meta-level MDPs are notoriously difficult to solve due to their extremely
large state space (Consul et al., 2022, Hay et al., 2014). In the project
selection task, the meta-level MDP has $(max_{obs}-min_{obs}+2)^{N_{P}\cdot
N_{C}\cdot N_{E}}$ possible belief states and $N_{P}\cdot N_{C}\cdot N_{E}+1$
possible meta-level actions. Our meta-level MDP introduces multiple new
intricacies that current metareasoning methods like MGPO (Heindrich et al.,
2023) aren’t equipped to handle, making strategy discovery in this setting
especially difficult. Compared to previously formulated meta-level MDPs
(Callaway et al., 2018a, Hay et al., 2014, Consul et al., 2022, Heindrich et
al., 2023, Mehta et al., 2022), our meta-level description of project
selection differs in the following ways: (1) the maximum amount of meta-level
actions is constrained with a budget, (2) the project selection task features
multiple consultants who differ in both the quality of their advice and the
cost of their services, (3) consultants in the project selection task offer
discrete estimates of each criterion , requiring that (4) criteria ratings are
scaled to allow weighting the criteria according to their importance.

##  4 A new metareasoning algorithm for discovering decision strategies for
human project selection

Previous metareasoning methods are unable to handle some of the intricacies of
the project selection problem. Therefore, we developed a new strategy
discovery method based on MGPO (Heindrich et al., 2023), which overcomes the
limitations that prevent MGPO from being applicable to project selection. To
reflect the commonalities and innovations, we call our new strategy discovery
method MGPS (meta-greedy policy for project selection). Similar to MGPO, our
method approximates the value of computation (VOC) (Russell and Wefald, 1991)
by myopically estimating the immediate improvement in decision quality.
Improving upon MGPO, MGPS calculates the myopic approximation to the VOC in a
way that accounts for discrete criteria ratings, criteria scaling, and
multiple sources of information with different costs and reliabilities.

Algorithm 1 MGPS VOC calculation for an action $a_{p_{i},c_{i},e_{i}}$

1:function myopic_voc($p_{i}$, $c_{i}$, $e_{i}$, $b$)

2: $r_{p}\leftarrow\mathbb{E}[r^{O}(p_{i})]$

3:
$r_{alt}\leftarrow\max_{p_{j}\in\mathcal{P}-\\{p_{i}\\}}\mathbb{E}[r^{O}(p_{j})]$

4: $\mu$, $\sigma\leftarrow b_{p_{i}c_{i}}$

5: for $obs$ from $min_{obs}$ to $max_{obs}$ do

6: if $min_{obs}<obs<max_{obs}$ then

7:
$p_{obs}\leftarrow\Phi\left(\frac{w_{c_{i}}\cdot(obs+0.5-\mu)}{\sqrt{(w_{c_{i}}\cdot\sigma)^{2}+(w_{c_{i}}\cdot\sigma_{e})^{2}}}\right)-\Phi\left(\frac{w_{c_{i}}\cdot(obs-0.5-\mu)}{\sqrt{(w_{c_{i}}\cdot\sigma)^{2}+(w_{c_{i}}\cdot\sigma_{e})^{2}}}\right)$

8: else if $obs==min_{obs}$ then

9:
$p_{obs}\leftarrow\Phi\left(\frac{w_{c_{i}}\cdot(obs+0.5-\mu)}{\sqrt{(w_{c_{i}}\cdot\sigma)^{2}+(w_{c_{i}}\cdot\sigma_{e})^{2}}}\right)$

10: else

11: $p_{obs}\leftarrow
1-\Phi\left(\frac{w_{c_{i}}\cdot(obs-0.5-\mu)}{\sqrt{(w_{c_{i}}\cdot\sigma)^{2}+(w_{c_{i}}\cdot\sigma_{e})^{2}}}\right)$

12: end if

13:
$\hat{\mu}_{obs}\leftarrow\left(\frac{w_{c_{i}}\cdot\mu}{(w_{c_{i}}\cdot\sigma)^{2}}+\frac{w_{c_{i}}\cdot
obs}{(w_{c_{i}}\cdot\sigma_{e})^{2}}\right)\cdot\left((w_{c_{i}}\cdot\sigma)^{2}+(w_{c_{i}}\cdot\sigma_{e})^{2}\right)$

14: end for

15: if $r_{p}>r_{alt}$ then

16: voc $\leftarrow\sum_{obs=min_{obs}}^{max_{obs}}p_{obs}(r_{p_{alt}}+\mu-
r_{p}-\hat{\mu}_{obs})\cdot\mathds{1}(r_{p}-\mu+\hat{\mu}_{obs}<r_{alt})$

17: else

18: voc
$\leftarrow\sum_{obs=min_{obs}}^{max_{obs}}p_{obs}(r_{p}+\hat{\mu}_{obs}-\mu-
r_{p_{alt}})\cdot\mathds{1}(r_{p}-\mu+\hat{\mu}_{obs}>r_{alt})$

19: end if

20: return $(1-w_{\lambda})$voc - $w_{\lambda}\lambda_{e_{i}}$

21:end function

MGPS calculates a myopic approximation to the VOC of asking an expert about a
specific criterion of a single project according to Algorithm 1. To account
for discrete advisor outputs, Algorithm 1 iterates over the discrete set of
possible ratings the expert might give and estimates the probability $p_{obs}$
of each rating ($obs$) and the belief update that would result from it
$\hat{\mu}_{obs}$ . The probability of each rating is computed using the
cumulative distribution function ($\Phi$) of the belief state for the selected
project’s criterion score (see Line 7). Here, the standard deviation
$\sigma_{e}$ of the likelihood function encodes the expert’s reliability, the
prior ($\mathcal{N}(\mu,\sigma)$) is the current belief about the project’s
score on the evaluated criterion, and the weights $w_{c_{i}}$ convert the
criteria into a common currency. The belief update that would result from the
observation ($\hat{\mu}_{obs}$) is computed by applying the belief state
update (see Line 13 and Equation 3). For the highest and lowest possible
ratings, we instead calculate $p_{obs}$ using the open interval (see Lines 9
and 11). The updated expected value of the belief state according to an
observation $obs$ is then calculated using Bayesian inference to integrate the
new observation into the belief state (see Line 13).

The VOC calculation depends on the posterior belief states that would result
from the different possible observations ($\hat{\mu}_{obs}$), weighted by
their probabilities. If the evaluated project currently has the highest
expected reward (see Line 15), the VOC calculation expresses the probability
of observing a value low enough that the second-best project becomes the most
promising option (see Line 16). In the case where the evaluated project did
not have the highest expected termination reward, the VOC calculation
expresses the probability of observing a value high enough to make the
evaluated project the most promising option (see Line 18). Finally, the cost
of the requested expert $\lambda_{e_{i}}$ is weighted using a free cost weight
parameter $w_{\lambda}$ and subtracted from the VOC estimate (see Line 20).

The full meta-greedy policy consists of calculating the VOC for all possible
meta-level actions and iteratively selecting the meta-level action with the
highest VOC. If no action has a positive VOC, the termination action $\perp$
is chosen.

##  5 Improving human project selection

Having developed a general metareasoning method for discovering resource-
rational strategies for human project selection, we now extend it to an
intelligent cognitive tutor for teaching people how to select better projects.
Our goal is to evaluate whether humans can utilize the strategies discovered
by MGPS and to provide a proof of concept for a general AI-powered boosting
approach that can be used to improve human decision-making across a wide range
of project selection problems. We first introduce a general approach for
teaching people the project selection strategies discovered by MGPS, and then
apply it to a real-world project selection problem.

###  5.1 MGPS Tutor: An intelligent tutor for teaching people how to select
better projects

Figure 1: Example of the MGPS tutor offering a choice between requesting
information from three different experts (highlighted in orange) in the
simplified training task of deciding between two project alternatives.

Our intelligent tutor for project selection (MGPS Tutor) trains people to
select the near-optimal decision operations identified by MGPS. To achieve
this, it lets people practice on a series of project selection problems and
gives them feedback. MPGS Tutor leverages MPGS in two ways: i) to
pedagogically construct the set of queries the learner is asked to choose
from, and ii) to give the learner feedback on their chosen query.

Building on the choice tutor by (Heindrich et al., 2023), our tutor repeatedly
asks the learner to choose from a pedagogically chosen set of decision
operations (see Figure 1) that always includes the query that MGPS would have
performed. Moreover, it leverages MGPS’s VOC calculation (Algorithm 1) to
score the chosen query, and then provides binary feedback on its quality. If
learners select a suboptimal expert, project, or criterion, they receive
feedback indicating the correct choice and have to wait for a short time. The
unpleasantness of having to wait serves as a penalty (Callaway et al., 2022).
Otherwise, they receive positive reinforcement and the next choice is
displayed. To receive positive reinforcement, the learner must select a query
whose VOC is sufficiently close, as determined by a tolerance parameter $t$,
to the VOC of the optimal action. We set the tolerance to $t=0.001$.

Our tutor teaches the strategy discovered by MGPS using a novel sophisticated
training schedule, which fosters learning by incrementally increasing the
complexity of the training task. This learning methodology is also known as
shaping (Skinner, 1953), and has been successfully applied to teach decision
strategies to humans (Heindrich et al., 2023). Our training schedule varies
the numbers of projects, how many different expert assessments learners have
to choose between, and the specific types of expert assessments offered as
choices. In total, our tutor teaches the discovered project selection strategy
using ten training trials. The first seven training trials use a smaller
version of the project selection task with only two projects, while the last
three trials use the full environment with five projects. The number of
choices gradually increases throughout training from 1 in the first training
trial to 9 in the last three training trials. The tutor varies the types of
choices across trials. After an initial trial with only a single choice, the
tutor offers choices that focus on different criteria within the same project
for two trials. Then, the tutor offers choices that focus on different experts
within the same project for two trials. The remaining trials combine both
types of highlights while sometimes also featuring queries about different
projects and also increasing the overall number of choices.

###  5.2 Evaluating the effectiveness of MGPS Tutor in a training experiment

To evaluate if AI-powered boosting can improve human project selection, we
tested the MPGS tutor in a training experiment. We tested if people trained by
MPGS tutor learn more resource-rational project selection strategies. To make
our assessment task as naturalistic as possible, we modelled it on a real
project selection problem that was faced by an Iranian financial institution
(Khalili-Damghani and Sadi-Nezhad, 2013). We will first describe how we
modeled this real-world problem, and then the training experiment.

#### A project-selection problem from the real world

Khalili-Damghani and Sadi-Nezhad (2013) worked on the real-world problem of
helping a financial institution select between five potential projects with an
eye to sustainability. Each project was evaluated by six advisors, who
assigned scores from one to five on six different criteria. For our model of
the task, we use the same number of experts, criteria, and projects, and the
same criteria weights as the financial institution. The remaining parameters
of the meta-level MDP were estimated as follows. We initialized the beliefs
about the project’s attributes by calculating the mean and the standard
deviation of all expert ratings for each criterion according to (Khalili-
Damghani and Sadi-Nezhad, 2013). We estimated the reliability of each expert
by calculating the standard deviation from the average distance of their
ratings from the average rating of all other experts, weighted by the number
of occurrences of each guess. We estimated the cost parameter $\lambda$ of the
meta-level MDP by $\lambda=\frac{cost}{stakes}\cdot r(\perp)$ to align the
meta-level MDP’s cost-reward ratio to its real-world equivalent. Using the
expected termination reward of the environment $r(\perp)=3.4$ and rough
estimates for the stakes $stakes=\$10000000$ and expert costs $cost=\$5000$,
this led to $\lambda=0.002$. While Khalili-Damghani and Sadi-Nezhad (2013)
assumed all expert ratings are available for free, in the real world this is
rarely the case. To make our test case more representative, we assumed that
advisor evaluations are available on-request for a consulting fee. To capture
that real-world decisions often have deadlines that limit how much information
can be gathered, we set the maximum number of sequentially requested expert
consultations to $5$.

#### Methods of the experiment

We recruited 301 participants for an online training experiment on Prolific.
The average participant age was 29 years, and 148 participants identified as
female. Participants were paid £3.50 for completing the experiment, plus an
average bonus of £0.50. The median duration of the experiment was 22 minutes,
resulting in a median pay of £10.9 per hour. Our experiment was preregistered
on AsPredicted and approved by the ethics commission of the Medical Faculty of
the University of Tübingen under IRB protocol number 667/2018BO2.

Each participant was randomly assigned to one of three conditions: (1) the No
tutor condition, in which participants did not receive any feedback and were
free to discover efficient strategies on their own; (2) the MGPS tutor
condition, in which participants practiced with our cognitive tutor that
provided feedback on the resource-rationality score MGPS assigns to the
selected planning actions; and (3) the Dummy tutor condition, an additional
control condition in which participants practiced with a dummy tutor
comparable to the MGPS tutor, albeit with randomized feedback on which
planning actions are correct. All participants practiced their planning
strategy in 10 training trials and were then evaluated across 10 test trials.
For each trial, a new instance of the project selection environment was
randomly generated by first sampling the project’s criteria scores from the
initial belief state and then generating each expert’s ratings based on their
reliability $\sigma_{e}$ using a Gaussian distribution centered at the
criteria score.

We evaluated the participants’ performance using two measures: their RR-score
and click agreement (Heindrich et al., 2023). RR-score’s are normalized by
subtracting the average reward of a random baseline algorithm and dividing by
the participant scores’ standard deviation. The random baseline algorithm is
defined as the policy that chooses meta-level actions at random until the
maximum number of decision operations is reached. Click agreement measures,
how well participants learned to follow the near-optimal strategy discovered
by our method. Specifically, we computed for each participant, which
proportion of their information requests matched the action taken by the
approximately resource-rational strategy discovered by MGPS.

#### Experiment results

Condition | RR-score | Click Agreement
---|---|---
MGPS Tutor | $0.3256\pm 0.0609$ | $0.4271\pm 0.0201$
No tutor | $-0.0227\pm 0.0622$ | $0.2521\pm 0.0171$
Dummy tutor | $0.0225\pm 0.0612$ | $0.2664\pm 0.0159$
Table 1: Results of the human training experiment. Per condition, the
normalized mean resource-rationality score and the mean click agreement are
reported. For both measures, we also report the 95% confidence interval under
the Gaussian assumption ($\pm 1.96$ standard errors).

Table 1 shows the results of the experiment. To determine whether the
condition of participants had a significant effect on the RR-score and click
agreement, we used an ANOVA analysis with Box approximation (Box et al.,
1954). The ANOVA revealed a significant effect of condition on both RR-score
($F(1.99,293.57)=10.48$, $p<.0001$) and click agreement
($F(1.99,291.48)=15.5$, $p<.0001$). We further compared the performance of
participants in the _MGPS tutor_ condition to participants in the two control
conditions with post hoc ANOVA-type statistics and used Cohen’s d (Cohen,
2013) to evaluate the size of the effects. The post hoc tests revealed that
participants in the _MGPS tutor_ condition achieved a significantly higher RR-
score than participants in the _No tutor_ ($F(1)=17.88$, $p<.0001$, $d=.35$)
and _Dummy tutor_ ($F(1)=13.4$, $p=.0002$, $d=.31$) conditions. Additionally,
participants in the _MGPS tutor_ reached a higher click agreement with our
pre-computed near-optimal strategy than participants in the _No tutor_
($F(1)=25.08$, $p<.0001$, $d=.58$) and _Dummy tutor_ ($F(1)=19.3$, $p<.0001$,
$d=.56$) conditions.

When evaluated on the same test trials and normalizing against the baseline
reward and the standard deviation of the experiment results, MGPS achieves a
mean reward of $1.17$, demonstrating that MGPS discovers more resource-
rational strategies than participants across all conditions. Although
participants in the MGPS tutor condition performed significantly the better
than participants in the other conditions, they did not learn to follow the
strategy taught by the tutor exactly. Participants in the other conditions
only discovered strategies with a similar RR-score to the random baseline
strategy, with participants in the No tutor condition performing even worse
than the random baseline strategy, and participants in the Dummy tutor
condition outperforming the random baseline only by a small margin.

#### What strategy did the MGPS Tutor teach?

To understand what participants in the MGPS Tutor condition were taught, we
inspected the strategy discovered by MGPS. We found that this strategy
systematically asks the most reliable experts ($e_{2}$ and $e_{6}$) to
evaluate projects on the criterion with the highest decision weight ($c_{5}$).
It starts by asking expert $e_{2}$ to rate criterion $c_{5}$ for a randomly
selected project. If that rating is below the maximum score, the same expert
is immediately asked to rate the same criterion for a different project. If
expert $e_{2}$ gives project $p_{i}$ the highest possible score on criterion
$c_{5}$ ($max_{obs}=5$), the strategy requests a second opinion about it from
expert $e_{6}$. If expert $e_{6}$ also evaluates the criterion’s value as $5$,
the decision process ends and project $p_{i}$ is selected. If expert $e_{6}$
provides a rating below $5$, expert $e_{2}$ is asked about a different
project. This process is repeated until a project in which both experts
$e_{2}$ and $e_{6}$ estimate $c_{5}=5$ is found, or the maximum number of
decision operations ($5$) is reached. It is important to note that while this
decision strategy is relatively easy to understand, describe, and execute, the
task of discovering this specific strategy as more resource-rational than the
vast number of possible other strategies is considerably more difficult.

#### What aspects of the strategy did participants learn?

We further analyzed which aspects of MGPS’ strategy participants learned from
the intelligent tutor. Specifically, we investigated in which proportion of
the test trials participants managed to (1) start planning with an optimal
action and (2) correctly decide whether to continue investigating the same
project or switch to requesting information about an alternative project.
MGPS’ strategy starts by requesting information about criterion $c_{5}$ from
one of the two most reliable experts. Participants in the MGPS tutor condition
did so $56\%$ of the time, whereas participants in the No tutor condition did
so only $33\%$ of the time. Looking into why many failed to request this
information, we found that more participants learned to request information
from the most reliable experts ($73\%$ for participants in the MGPS tutor
condition and $43\%$ for participants in the No tutor condition) than to
request information about the most important evaluation criteria ($64\%$ for
participants in the MGPS tutor condition and $43\%$ for participants in the No
tutor condition).

A second important aspect of MGPS’s strategy is how it responds to the
information revealed. Excluding trials in which participants’ first expert
request did not match an action MGPS identifies as optimal, we analyzed
whether participants either correctly continued to evaluate the current
project (in case of revealing an expert guess of $5$), or correctly switched
to evaluating an alternative project (when revealing an expert guess smaller
than $5$). Participants in the MGPS tutor condition ($65\%$ of test trials)
and participants in the No tutor condition ($65\%$ of test trials) both
learned to accurately switch projects when appropriate. Repeatedly evaluating
the current project proved to be the most difficult component of the
discovered strategy to learn, with participants in the only MGPS tutor
condition following this aspect of the strategy in $11\%$ of test trials and
participants of the No tutor condition following the strategy in only $7\%$ of
test trials.

##  6 Performance evaluation

The results reported in the previous section show that MPGS can discover
project selection strategies that are more effective than the strategies
people discover on their own. But how does its performance compare to other
strategy discovery algorithms? To answer this question, we evaluated MGPS on a
computational benchmark. We chose PO-UCT (Silver and Veness, 2010) for
comparisons because it is an established baseline for metareasoning algorithms
in partially observable environments (Heindrich et al., 2023) and the more
specialized MGPO algorithm is not applicable to project selection. PO-UCT
utilizes Monte Carlo tree search to simulate the effects of different actions,
resulting in more accurate results with increased computation time, making it
a useful baseline for MGPS’s computational efficiency and performance.

#### Method

We evaluated the effectiveness of our method in the project selection task by
comparing it against PO-UCT (Silver and Veness, 2010) with different numbers
of steps. All methods were evaluated across the same 5000 randomly generated
instances of the project selection environment.

Our main performance measure was the resource-rationality score (RR-Score
defined in Equation 1). To highlight the achieved improvements over a baseline
algorithm that performs random meta-level actions, we normalized the reported
RR-scores. Specifically, we applied a z-score transformation, subtracting the
average reward of the random baseline algorithm (see Section 5.2) from the RR-
Scores and dividing by the evaluated algorithm’s RR-Scores’ standard
deviation. We analyze the differences in RR-Scores with an ANOVA and evaluate
the size of statistical effects with Cohen’s d (Cohen, 2013). Additionally, we
compare the computational efficiency of the different methods, which is
crucial for being able to provide real-time feedback in our cognitive tutor.

#### Results

Algorithm | RR-score | Runtime (s)
---|---|---
MGPS | $0.9942\pm 0.0234$ | $0.9079\pm 0.0052$
PO-UCT (10 steps) | $-0.4344\pm 0.0106$ | $0.0175\pm 0.0004$
PO-UCT (100 steps) | $0.7309\pm 0.0302$ | $0.1972\pm 0.0008$
PO-UCT (1000 steps) | $0.8681\pm 0.0256$ | $2.3567\pm 0.0028$
PO-UCT (5000 steps) | $0.9054\pm 0.0232$ | $10.8913\pm 0.0173$
Table 2: Results of the performance evaluation. For each algorithm, we report
the averge normalized resource-rationality score (RR-Scores) and the runtime
per decision problem. For both measures, we also report the 95% confidence
interval under the Gaussian assumption ($\pm 1.96$ standard errors).

As shown in Table 2, MGPS outperformed all tested versions of PO-UCT and the
random baseline strategy (as the RR-scores are normalized by subtracting the
mean RR-score of the random baseline, the random baseline strategy itself has
a normalized RR-score of $0$). An ANOVA revealed significant differences in
the RR-scores of the strategies discovered by the different methods
($F(4,24995)=2447$, $p<.0001$). Hukey-HSD post-hoc comparisons indicated that
the strategies discovered by MGPS are significantly more resource-rational
than the strategies discovered by PO-UCT with 10 steps ($p<.0001$, $d=2.18$),
100 steps ($p<.0001$, $d=.27$), 1000 steps ($p<.0001$, $d=.14$), or 5000 steps
($p<.0001$, $d=.11$). While MGPS achieves significantly higher RR-scores than
all PO-UCT variants, the size of the effect decreases from a very large effect
to a small effect when increasing PO-UCT’s computational budget sufficiently.
We therefore expect that PO-UCT with a much more than 5000 steps would
ultimately achieve comparable RR-scores to MGPO, albeit at a much higher
computational cost. Moreover, MGPS was substantially faster than PO-UCT with a
computational budget of 1000 steps or more, which is when PO-UCT’s performance
starts to approach that of MGPS. With a computational budget of 100 steps or
fewer, PO-UCT is faster than MGPS. However, such a small computational budget
is not enough for PO-UCT to discover strategies with a RR-score anywhere near
that of the strategy discovered by MGPS. Critically, the high amount of
computation required for PO-UCT to achieve an approximately similar level of
resource-rationality would render PO-UCT unusable for a cognitive tutor that
computes feedback in real time.

##  7 Conclusion

People’s decision-making is prone to systematic errors (Kahneman et al.,
1982), and although people are happy to delegate some decisions, most CEOs are
unlikely to let AI decide which projects their company should pursue.
Moreover, people are reluctant to use normative project selection procedures
because they tend to be more tedious (Schmidt and Freeland, 1992, Liu et al.,
2017, Kornfeld and Kara, 2013). Motivated by people’s insistence on making
their own decisions with simple heuristics, we leveraged AI to discover and
teach prescriptive decision strategies adapted to human cognition that perform
substantially better than people’s intuitive strategies but are nevertheless
simple enough that people use them. To this end, we introduced a metareasoning
method for leveraging AI to discover near-optimal decision strategies for
human project selection. Modeling project selection through the lens of
resource rationality allowed us to formulate a mathematically precise
criterion for the quality of decision strategies for human project selection.
We further developed an efficient automatic strategy discovery algorithm
automatically discovers efficient strategies for human project selection. Our
algorithm discovered decision strategies that are much more resource-rational
than the strategies humans discovered on their own and the strategies
discovered by a general-purpose algorithm for solving POMDPs (PO-UCT). Using
the decision strategies discovered by our algorithm, we created a cognitive
tutor that uses a shaping schedule and metacognitive feedback to teach the
strategies to humans. In the training experiment, our cognitive tutor fostered
significant improvements in participants’ resource rationality.

A main limitation of our method is that it is unknown how precisely the
environment parameters need to be estimated to construct the metareasoning
task, which can prove especially problematic when there isn’t much data on
past decisions. Future work could investigate and potentially address this
issue by extending MGPS with a Bayesian inference approach to estimate
uncertain parameters of the environment (Mehta et al., 2022). This extension
would make MGPS robust to uncertainties in the underlying distributions of
evaluation criteria or expert reliability, and make it possible to discover
resource-rational strategies even when data to estimate environment parameters
is limited.

Encouraged by the promising results from successfully teaching humans in our
naturalistic model of project selection, we are excited about future work
assessing the real-world impact of improving people’s decision-making by
evaluating their decisions directly in the real world. Relevant real-world
decisions could be a company deciding which research project to invest in, or
a charity deciding which initiatives maximize impact and cost-effectiveness.
Similarly, while our cognitive tutor proved to be effective in teaching humans
critical aspects of MGPS’ discovered strategies, we are looking forward to
future pedagogical work that improves and further evaluates the mechanisms by
which the cognitive tutor teaches strategies. Additionally, we are also
excited about potential future work that combines MGPS with AI-Interpret
(Skirzyński et al., 2021) to automatically derive human-legible
recommendations for how to make project selection decisions. Lastly, although
MGPS performed very well on our benchmarks, MGPS’s myopic approximation could
fail in more complicated scenarios with interdependent criteria. Such
challenges could be addressed by solving meta-level MDPs with methods from
deep reinforcement learning, for example, by utilizing AlphaZero (Silver et
al., 2017).

Our results indicate that it is possible to use resource-rational analysis
combined with automatic strategy discovery to improve human decision-making in
a realistic project selection problem. As selecting projects is a common
problem faced by both organizations and individuals, improving their decision
strategies in this setting would have a direct positive impact. Specialized
project selection tutors for specific types of project selection problems
could be used to train current and future leaders within companies and
organizations. Moreover, project-selection tutors could be integrated into MBA
programs to teach future decision-makers efficient decision strategies as part
of their education. We are optimistic that our general methodology is also
applicable to other real-world problems, offering a promising pathway to teach
people efficient strategies for making better decisions in other areas as
well. Besides project selection problems, we believe our approach could be
used to improve real-world decision-making in areas such as grant-making and
policy decisions.

### Acknowledgements

This project was funded by grant number CyVy-RF-2019-02 from the Cyber Valley
Research Fund.

The authors thank the International Max Planck Research School for Intelligent
Systems (IMPRS-IS) for supporting Lovis Heindrich.

The code of the strategy discovery method as well as data and analysis for the
simulation and training experiment are available online:
https://github.com/lovis-heindrich/MGPS-project-selection/.

## References

  * Abdel-Basset et al. (2019) M. Abdel-Basset, A. Atef, and F. Smarandache.  A hybrid neutrosophic multiple criteria group decision making approach for project selection.  _Cognitive Systems Research_ , 57:216–227, 2019.
  * Becker et al. (2022) F. Becker, J. Skirzyński, B. van Opheusden, and F. Lieder.  Boosting human decision-making with ai-generated decision aids.  _Computational Brain & Behavior_, pages 1–24, 2022.
  * Bonaccio and Dalal (2006) S. Bonaccio and R. S. Dalal.  Advice taking and decision-making: An integrative literature review, and implications for the organizational sciences.  _Organizational behavior and human decision processes_ , 101(2):127–151, 2006.
  * Boutilier (2002) C. Boutilier.  A pomdp formulation of preference elicitation problems.  In _AAAI/IAAI_ , pages 239–246. Edmonton, AB, 2002.
  * Box et al. (1954) G. E. Box et al.  Some theorems on quadratic forms applied in the study of analysis of variance problems, i. effect of inequality of variance in the one-way classification.  _The annals of mathematical statistics_ , 25(2):290–302, 1954.
  * Callaway et al. (2018a) F. Callaway, S. Gul, P. M. Krueger, T. L. Griffiths, and F. Lieder.  Learning to select computations.  _Uncertainty in Artificial Intelligence_ , 2018a.
  * Callaway et al. (2018b) F. Callaway, F. Lieder, P. Das, S. Gul, P. M. Krueger, and T. Griffiths.  A resource-rational analysis of human planning.  In _CogSci_ , 2018b.
  * Callaway et al. (2022) F. Callaway, Y. R. Jain, B. van Opheusden, P. Das, G. Iwama, S. Gul, P. M. Krueger, F. Becker, T. L. Griffiths, and F. Lieder.  Leveraging artificial intelligence to improve people’s planning strategies.  _Proceedings of the National Academy of Sciences of the United States of America_ , Mar. 2022.  doi: 10.1073/pnas.2117432119.  URL https://www.pnas.org/doi/10.1073/pnas.2117432119.
  * Carazo et al. (2012) A. F. Carazo, I. Contreras, T. Gómez, and F. Pérez.  A project portfolio selection problem in a group decision-making context.  _Journal of Industrial & Management Optimization_, 8(1):243, 2012.  Publisher: American Institute of Mathematical Sciences.
  * Cohen (2013) J. Cohen.  _Statistical power analysis for the behavioral sciences_.  Academic press, 2013.
  * Coldrick et al. (2002) S. Coldrick, C. Lawson, P. Ivey, and C. Lockwood.  A decision framework for r&d project selection.  In _IEEE International Engineering Management Conference_ , volume 1, page 413–418. IEEE, 2002.
  * Consul et al. (2022) S. Consul, L. Heindrich, J. Stojcheski, and F. Lieder.  Improving human decision-making by discovering efficient strategies for hierarchical planning.  _Computational Brain & Behavior_, 5(2):185–216, 2022.
  * de Souza et al. (2021) D. G. B. de Souza, E. A. dos Santos, N. Y. Soma, and C. E. S. da Silva.  MCDM-Based R&D Project Selection: A Systematic Literature Review.  _Sustainability_ , 13(21):11626, 2021.  Publisher: MDPI.
  * Doshi and Roy (2008) F. Doshi and N. Roy.  The permutable pomdp: fast solutions to pomdps for preference elicitation.  In _Proceedings of the 7th international joint conference on Autonomous agents and multiagent systems-Volume 1_ , pages 493–500, 2008.
  * Eom and Kim (2006) S. Eom and E. Kim.  A survey of decision support system applications (1995–2001).  _Journal of the Operational Research Society_ , 57:1264–1278, 2006.
  * Gino (2008) F. Gino.  Do we listen to advice just because we paid for it? the impact of advice cost on its use.  _Organizational Behavior and Human Decision Processes_ , 107:234–245, 2008.
  * Gupta et al. (2022) S. Gupta, S. Modgil, S. Bhattacharyya, and I. Bose.  Artificial intelligence for decision support systems in the field of operations research: review and future scope of research.  _Annals of Operations Research_ , pages 1–60, 2022.
  * Hay et al. (2014) N. Hay, S. Russell, D. Tolpin, and S. E. Shimony.  Selecting computations: Theory and applications.  _arXiv preprint arXiv:1408.2048_ , 2014.
  * Heindrich et al. (2023) L. Heindrich, S. Consul, and F. Lieder.  Leveraging ai to improve human planning in large partially observable environments.  _arXiv preprint arXiv:2302.02785_ , 2023.
  * Henriksen and Traynor (1999) A. Henriksen and A. Traynor.  A practical r&d project-selection scoring tool.  _IEEE Transactions on Engineering Management_ , 46(2):158–170, 1999.  doi: 10.1109/17.759144.
  * Hertwig and Grüne-Yanoff (2017) R. Hertwig and T. Grüne-Yanoff.  Nudging and boosting: Steering or empowering good decisions.  _Perspectives on Psychological Science_ , 12(6):973–986, 2017.
  * Kahneman et al. (1982) D. Kahneman, S. P. Slovic, P. Slovic, and A. Tversky.  _Judgment under uncertainty: Heuristics and biases_.  Cambridge university press, 1982.
  * Khalili-Damghani and Sadi-Nezhad (2013) K. Khalili-Damghani and S. Sadi-Nezhad.  A hybrid fuzzy multiple criteria group decision making approach for sustainable project selection.  _Applied Soft Computing_ , 13(1):339–352, 2013.
  * Kornfeld and Kara (2013) B. Kornfeld and S. Kara.  Selection of Lean and Six Sigma projects in industry.  _International Journal of Lean Six Sigma_ , 2013.  Publisher: Emerald Group Publishing Limited.
  * Lieder and Griffiths (2020) F. Lieder and T. L. Griffiths.  Resource-rational analysis: understanding human cognition as the optimal use of limited computational resources.  _Behavioral and Brain Sciences_ , 43, 2020.
  * Lieder et al. (2019) F. Lieder, F. Callaway, Y. Jain, P. Krueger, P. Das, S. Gul, and T. Griffiths.  A cognitive tutor for helping people overcome present bias.  In _RLDM 2019_ , 2019.
  * Liu et al. (2017) F. Liu, W.-d. Zhu, Y.-w. Chen, D.-l. Xu, and J.-b. Yang.  Evaluation, ranking and selection of R&D projects by multiple experts: an evidential reasoning rule based approach.  _Scientometrics_ , 111(3):1501–1519, 2017.  Publisher: Springer.
  * Mehta et al. (2022) A. Mehta, Y. R. Jain, A. Kemtur, J. Stojcheski, S. Consul, M. Tošić, and F. Lieder.  Leveraging machine learning to automatically derive robust decision strategies from imperfect knowledge of the real world.  _Computational Brain & Behavior_, 5(3):343–377, 2022.
  * Mohagheghi et al. (2019) V. Mohagheghi, S. M. Mousavi, J. Antuchevičienė, and M. Mojtahedi.  Project portfolio selection problems: a review of models, uncertainty approaches, solution techniques, and case studies.  _Technological and Economic Development of Economy_ , 25(6):1380–1412, 2019.
  * Olsen et al. (2019) K. Olsen, A. Roepstorff, and D. Bang.  Knowing whom to learn from: individual differences in metacognition and weighting of social information.  _PsyArXiv_ , 2019.
  * Ronayne et al. (2019) D. Ronayne, D. Sgroi, et al.  _Ignoring good advice_.  University of Warwick, Centre for Competitive Advantage in the Global …, 2019.
  * Russell and Wefald (1991) S. Russell and E. Wefald.  Principles of metareasoning.  _Artificial intelligence_ , 49(1–3):361–395, 1991.
  * Sadi-Nezhad (2017) S. Sadi-Nezhad.  A state-of-art survey on project selection using mcdm techniques.  _Journal of Project Management_ , 2(1):1–10, 2017.
  * Schmidt and Freeland (1992) R. L. Schmidt and J. R. Freeland.  Recent progress in modeling R&D project-selection processes.  _IEEE Transactions on Engineering Management_ , 39(2):189–201, 1992.  Publisher: IEEE.
  * Silver and Veness (2010) D. Silver and J. Veness.  Monte-carlo planning in large pomdps.  _Advances in neural information processing systems_ , 23, 2010.
  * Silver et al. (2017) D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, T. P. Lillicrap, K. Simonyan, and D. Hassabis.  Mastering chess and shogi by self-play with a general reinforcement learning algorithm.  _CoRR_ , abs/1712.01815, 2017.  URL http://arxiv.org/abs/1712.01815.
  * Skinner (1953) B. Skinner.  _Shaping and maintaining operant behavior_ , pages 91–106.  Free Press New York, 1953.
  * Skirzyński et al. (2021) J. Skirzyński, F. Becker, and F. Lieder.  Automatic discovery of interpretable planning strategies.  _Machine Learning_ , 110:2641–2683, 2021.
  * Yaniv and Kleinberger (2000) I. Yaniv and E. Kleinberger.  Advice taking in decision making: Egocentric discounting and reputation formation.  _Organizational behavior and human decision processes_ , 83(2):260–281, 2000.