# The impact of surplus sharing on the outcomes of specific investments under
negotiated transfer pricing

An agent-based simulation with fuzzy Q-learning agents

Christian Mitsch Christian Mitsch (0000-0002-8587-4887) Department of
Management Control and Strategic Management, University of Klagenfurt
Universitätsstraße 65-67, 9020 Klagenfurt, Austria
22email<EMAIL_ADDRESS>
###### Abstract

This paper focuses on specific investments under negotiated transfer pricing.
Reasons for transfer pricing studies are primarily to find conditions that
maximize the firm’s overall profit, especially in cases with bilateral trading
problems with specific investments. However, the transfer pricing problem has
been developed in the context where managers are fully individual rational
utility maximizers. The underlying assumptions are rather heroic and, in
particular, how managers process information under uncertainty, do not
perfectly match with human decision-making behavior. Therefore, this paper
relaxes key assumptions and studies whether cognitively bounded agents achieve
the same results as fully rational utility maximizers and, in particular,
whether the recommendations on managerial-compensation arrangements and
bargaining infrastructures are designed to maximize headquarters’ profit in
such a setting.

Based on an agent-based simulation with fuzzy Q-learning agents, it is shown
that in case of symmetric marginal cost parameters, myopic fuzzy Q-learning
agents invest only as much as in the classic hold-up problem, while non-myopic
fuzzy Q-learning agents invest optimally. However, in scenarios with non-
symmetric marginal cost parameters, a deviation from the previously
recommended surplus sharing rules can lead to higher investment decisions and,
thus, to an increase in the firm’s overall profit.

###### Keywords:

Agent-based simulation and Bounded rationality and Fuzzy Q-learning and Hold-
up problem and Negotiated transfer pricing and Specific investments

∎

The impact of surplus sharing on the outcomes of specific investments under
negotiated transfer pricing
An agent-based simulation with fuzzy Q-learning agents
$\text{Christian Mitsch}^{[0000-0002-8587-4887]}$

Department of Management Control and Strategic Management

University of Klagenfurt, 9020, Austria

<EMAIL_ADDRESS>
##  1 Introduction

###  1.1 Problem formulation

As a business organization grows and the degree of the headquarters’ task
increases, delegating authority over operating decisions to managers might
result in benefits, as managers, e.g., can use their superior knowledge to
achieve better solutions for the firm (Wagenhofer 1994). In addition, the
concept of profit centers can be applied to determine profit for internal
divisions of responsibility and for purposes of management control, e.g.,
optimal resource allocation and profitability (e.g., Küpper et al. 2013;
Milgrom and Roberts 1992). Divisions’ interests may be in conflict with the
goal of the headquarters and, furthermore, divisions that are treated as
profit centers often operate in a competitive environment within the firm,
e.g., competing for financial resources. Competitive behavior among profit
centers of a firm may increase the profit of some divisions but does not
necessarily maximize the headquarters’ profit and, especially, they neglect
the potentially negative consequences of their decisions for other divisions
and the entire firm (Göx and Schiller 2006).

The divisions’ coordination problem increases when there is an internal
transfer of goods and services from one division to another, e.g.,
transferring an intermediate product. Internal transfers are often performed
under conditions of asymmetric information (Baldenius 2000). For instance, one
division has better knowledge about the costs of manufacturing the
intermediate product, whereas another division has private information related
to the net revenues from selling the intermediate product. Therefore, transfer
pricing as an instrument for coordinating operational decisions can be applied
to manage potential conflicts and to guide the internal trade (Baldenius et
al. 1999). With a perfectly competitive market for the intermediate product,
the optimal transfer price corresponds to the external market price, as this
leads to efficient divisions’ decision-making (Cook 1955). If there is no
external market for the intermediate product because the product is highly
specialized, the headquarters could set the transfer price equal to the
marginal cost of the intermediate product (Hirshleifer 1956; Schmalenbach
1908/1909). However, large business organizations usually do not have the
opportunity, e.g., due to high costs or due to lack of information, to create
an optimal production program of the entire firm centrally and, in generally,
the divisions’ cost information are required to optimally set the transfer
price.

The headquarters’ task is made more difficult when the divisions can
additionally make specific investments that increase the value of internal
trade. The main problem with specific investments is that they are
irreversible and are of little or no value in the divisions’ external lines of
business (Edlin and Reichelstein 1995). For example, one division may incur an
upfront fixed cost, e.g., an investment in production technology that reduces
the variable costs of manufacturing intermediate products, whereas another
division may improve the net revenues by investing in marketing activities.
However, due to the fact that the divisions bear the full cost of their own
investments, but are evaluated according to their profits, each division is
prone to underinvestment. This problem is also known as the “hold-up” problem
(Schmalenbach 1908/1909; Williamson 1979, 1985).

The hold-up problem could be prevented by signing a complete contract that
contains specific clauses for all possible future events that may occur after
the investment decision (Göx and Schiller 2006). But in reality, it is
impossible to specify contract clauses for all anticipated eventualities. For
this reason, different transfer pricing mechanisms are designed which ensure
the optimal solution of the headquarters’ control problem.

The remainder of this paper is organized as follows: The next section of the
introduction covers a short literature review of negotiated transfer pricing
models, followed by a criticism of key assumptions required to solve such
models. Then, a series of model relaxations to make assumptions about human
decision-making behavior more realistic is given and the introduction
concludes with the research method to study the hold-up problem with
cognitively bounded agents. In Sec. 2, negotiated transfer pricing for
bilateral negotiation problems with specific investments is introduced and the
resulting solutions are discussed. Sec. 3 introduces the relaxed assumptions
and describes what the agents can and cannot do in the agentized version of
the negotiated transfer pricing model.111 “Agentized” is derived from
agentization and describes the process of turning a neoclassical model into an
agent-based model (e.g., Guerrero and Axtell 2011; Leitner and Behrens 2015).
In addition, Sec. 3 formalizes how fuzzy Q-learning works and gives a brief
introduction to the exploration policies which are used in the simulation
study. The parameter settings for the agent-based simulation are explained in
Sec. 4 and, in Sec. 5, the results of the agent-based simulation are presented
and discussed. Sec. 6 contains concluding remarks and suggests possible
directions for future research.

Please be aware that, in the following, the additional model designation
“specific investment” is sometimes omitted for the sake of brevity.

###  1.2 Brief literature review

The economic transfer pricing research is typically divided into market-based,
cost-based, and negotiated transfer pricing methods, which are usually based
on game-theoretic models.222 For a survey of the transfer pricing literature
see, e.g., Göx and Schiller (2006), Wagenhofer (1994), and, for a survey in
German see, e.g., Ewert and Wagenhofer (2014), Pfaff and Pfeiffer (2004),
Wagner (2008). Readers unfamiliar with agency theory and game theory should
refer to Laffont and Martimort (2009) and Osborne and Rubinstein (1994),
respectively. Since it is generally impossible to give a recommendation for
the “best” transfer pricing method, a variety of solutions to the transfer
pricing problem are offered (Göx and Schiller 2006). This paper focuses on
negotiated transfer pricing, as is a common way of setting the transfer price
in multidivisional firms (e.g., Cook 1955; Eccles 1985).333 In a negotiated
transfer pricing model, divisions’ decisions are free from headquarters
interventions. Such models stand in contrast to administered transfer pricing
models Eccles and White (1988).

The starting point of the research is the well-known negotiated transfer
pricing model by Edlin and Reichelstein (1995). The authors extend the
neoclassical model of Schmalenbach (1908/1909) and Williamson (1985) by
assuming that each division can simultaneously make a specific investment that
enhances the value of internal trade before the negotiation over transfer
price and transfer quantity takes place. Edlin and Reichelstein (1995) show
that efficient investments can be guaranteed, if (1) an upfront fixed-price
contract which determines the quantity of the intermediate product to be
transferred and the corresponding transfer price and (2) the opportunity to
renegotiate this contract once cost and revenue information are available to
both divisions. In particular, the transfer pricing scheme by Edlin and
Reichelstein seems to be robust in the sense that, for any combination of
divisions’ bargaining power, the hold-up problem can be solved. It should be
noted that their results depend on the main assumption that both divisions
have symmetric information about costs and revenues in their bilateral
negotiation.

In contrast to an upfront fixed-price contract between divisions, Pfeiffer
(2004) adds an additional phase to the sequence of events within the
negotiated transfer pricing model in which the headquarters may grant
individual divisions access to “pre-investment” information. This phase should
help the divisions to revise their beliefs about the realizations of costs and
revenues, before each division makes an investment decision independently of
each other. Pfeiffer shows that installing a pre-investment information system
has an ambiguous effect on the undertaken investments. On one hand, it can
mitigate the hold-up problem and even lead to efficient investments, but it
can also cause dysfunctionalities for the entire firm. For a perfect pre-
investment information system, Pfeiffer provides conditions for the class of
linear surplus sharing rules under which the headquarters’ profit is
maximized.

Finally, well-known contributions of negotiated transfer pricing under
conditions of asymmetric information are given, e.g, by Ronen and Balachandran
(1988) or by Vaysman (1998). For instance, Vaysman considers a negotiated
transfer pricing model in a multidivisional firm where each division has
better divisional information than the headquarters and the other division.
Vaysman’s analysis indicates that in scenarios with information asymmetry,
managerial-compensation arrangements and bargaining infrastructures can be
designed to maximize headquarters’ profit.

###  1.3 Criticism of model assumptions

However, the transfer pricing problem has been developed in the context where
divisions are fully individual rational utility maximizers. For instance,
Wagenhofer (1994) uses backward induction to compute the subgame perfect
equilibrium of his modified negotiated transfer pricing model, or Vaysman
(1998) applies the concept of perfect Bayesian equilibrium, developed by
Fudenberg and Tirole (1991), to extend the idea of subgame perfection to games
of incomplete information. The concept of subgame perfect equilibrium is
attractive, but it should be mentioned that it has been sometimes criticized
because it depends excessively on rationality (e.g., Aumann 1989; Laffont and
Martimort 2009). For example, how could a division in its investment decision
assume that the other division will behave optimally at the later negotiation
as needed by subgame perfect implementation? Such concepts require heroic
assumptions about the information the divisions have regarding, for example,
the structure of the whole game, the underlying utility functions, and the
knowledge about opponent’s behavior (Sebenius 1992), or certain conditions
about the probability distributions of their environment (Simon 1979). In
practice, these assumptions are often not met, but, to put it in the words of
Aumann (1989), “The common knowledge assumption underlies all of game theory
and much of economic theory.”.

Apart from this, one could also criticize that the expectations of how
divisions interact with each other in a multi-stage decision-making process
and, in particular, how they process information under uncertainty, do not
perfectly match with human decision-making behavior. The relevant literature
also suggests that the assumption of rational behavior of economic agents, as
it is common in neoclassical economics, is too strong (e.g., Arthur 1991;
Richiardi et al. 2006; Simon 1955). In particular, assumptions on rationality
are implausible in models that reflect human behavior and, therefore, most of
such models quickly become impractical and unrealistic (Young 2001).

###  1.4 Relaxation of assumptions and research method

Against this background, an agent-based variant of negotiated transfer pricing
is analyzed in which (1) divisions have no beliefs about the rationality of
other divisions nor how other divisions face their maximization problem, (2)
they have no common knowledge of the opponent’s utility function, and (3) they
are subject to cognitive limitations such as limits on the capacity of human
memory, restrictions on the speed of learning, and constraints in foresight.
To deal with the divisions’ bounded rationality, asymmetric information, and
cognitive limitations, a computer simulation with individual learning agents
is set up for the before-mentioned hold-up problem.

The learning method in this paper rests on reinforcement learning because
using reinforcement learning in an agent-based simulation may be more
realistic as reinforcement learning provides methods that are close to the way
animals and humans learn things (e.g., Collins 2019; Niv 2009; Sutton and
Barto 2018). In particular, fuzzy Q-learning, which is “technically” based on
dynamic programming and Monte Carlo methods, is used in this simulation study
to describe the before-mentioned cognitively bounded agents.444 The
combination of dynamic programming and Monte Carlo ideas is called temporal-
difference learning, a class of algorithms in reinforcement learning. Readers
unfamiliar with reinforcement learning methods should refer to Sutton and
Barto (2018). For agent-based simulations with Q-learning agents acting in an
economic context see, e.g., Dearden et al. (1998); Tesauro and Kephart (2002);
Waltman and Kaymak (2008) and with fuzzy Q-learning agents see, e.g., Kofinas
et al. (2018); Rahimiyan and Mashhadi (2006).

Since the model investigated here is characterized by a high degree of
heterogeneity with respect to the structure and the dynamic interactions
between agents and their environment, agent-based simulation seems to be an
appropriate research method to deal with the complexity of the research
problem. In addition, an agent-based simulation also allows to observe the
agents’ behavior as well as the system’s behavior on the macro-level in time,
which otherwise cannot be derived in relation to a “functional relationship”
from the individual behaviors of those agents (e.g., Epstein 2006; Wall 2016).

This paper enriches the research on negotiated transfer pricing by an agent-
based simulation with fuzzy Q-learning agents. In particular, this simulation
study aims to provide a contribution to the hold-up problem by answering
whether cognitively bounded agents achieve the same results as fully rational
utility maximizers and, especially, whether the recommendations on managerial-
compensation arrangements and bargaining infrastructures maximize
headquarters’ profits under more realistic assumptions.

##  2 Negotiated transfer pricing with fully rational agents

###  2.1 Organizational setting

This paper analyzes a decentralized firm consisting of one headquarters and
two divisions - the supplying division (the “upstream” division) and the
buying division (the “downstream” division). In line with the previous
literature (e.g., Eccles and White 1988; Edlin and Reichelstein 1995; Göx and
Schiller 2006; Pfeiffer and Wagner 2007; Vaysman 1998; Wagner 2008), it is
assumed that both divisions are organized as profit centers, i.e., each
division is evaluated on its own divisional profit, and that the headquarters
delegates to its divisions operating decisions like the amount of specific
investment, the level of the transfer price, and the amount of intermediate
products. Suppose that the supplying division purchases raw materials from a
commodity market in order to manufacture an intermediate product which is
transferred to the buying division.555 In negotiated transfer pricing models,
it is commonly assumed that the intermediate product is highly specialized
and, therefore, there is no external market for the intermediate product
(e.g., Anctil and Dutta 1999; Edlin and Reichelstein 1995; Wagner 2008). For
the sake of simplicity, it is assumed that one unit of raw material is
required to manufacture one unit of the intermediate product. On the other
side, the buying division refines the intermediate product, e.g., to improve
its quality, and, finally, sells it on an outlet market. Figure 1
schematically represents the negotiated transfer pricing model investigated
here.

Figure 1: Schematic representation of the decentralized firm. In addition, the
sequence of events within the negotiated transfer pricing model is
illustrated. Source: Based on Wagner (2008).

In the following, supplying division, buying division, and headquarters are
abbreviated to $S$, $B$, and $HQ$, respectively. The supplying division’s
costs of manufacturing $q\in\mathbb{R}^{+}$ units of the intermediate product
are given by

$C_{S}(q,\theta_{S},I_{S})=(\theta_{S}-I_{S})\;q\;,$ (1)

where $\theta_{S}\in\mathbb{R}^{+}$ is a state variable which reflects the
purchase price of raw materials and $I_{S}\in\mathbb{R}^{+}$ stands for the
amount of specific investment carried by the supplying division. In contrast,
the buying division’s net revenue is

$R_{B}(q,\theta_{B},I_{B})=(\theta_{B}-\frac{1}{2}\;b\;q+I_{B})\;q\;,$ (2)

where $\theta_{B}\in\mathbb{R}^{+}$ is another state variable which represents
the constant term in the inverse demand function for the selling product,
$b\in\mathbb{R}^{+}$ describes the slope of the inverse demand function, and
$I_{B}\in\mathbb{R}^{+}$ denotes the amount of specific investment carried by
the buying division. To keep the analysis simple and, especially, that the
negotiated transfer pricing model has a unique subgame perfect equilibrium, it
is assumed that the inverse demand function and the costs of production are
linear and, for linguistic reasons, the term “investment decision” or just
“investment” is used for $I_{j}$, $j\in\\{S,B\\}$, instead of “amount of
specific investment”.

It is assumed that the supplying division and the buying division have private
information about the commodity market and the outlet market, respectively,
which are embodied in the state variables $\theta_{S}$ and $\theta_{B}$. In
addition, supplying and buying division have information about the expected
value of the outlet market and commodity market, respectively. For the sake of
simplicity, the state variables $\theta_{S}$ and $\theta_{B}$ are
stochastically independent random variables. Further, the analysis assumes
that the headquarters cannot observe the state variables nor the undertaken
investments; the headquarters only knows the expected values of the state
variables. Furthermore, the headquarters’ accounting system receives costs and
revenues after the negotiation phase and, subsequently, the headquarters’
profit $\Pi_{HQ}$ is calculated.

$\Pi_{HQ}(q,\theta_{S},\theta_{B},I_{S},I_{B})=R_{B}(q,\theta_{B},I_{B})-C_{S}(q,\theta_{S},I_{S})-w_{S}(I_{S})-w_{B}(I_{B})$
(3)

Both the supplying division’s investments and the buying division’s
investments cause divisional investment costs (or divisional capital
expenditure) $w_{S}(I_{S})$ and $w_{B}(I_{B})$, respectively, which have a
quadratic cost structure

$w_{j}(I_{j})=\frac{1}{2}\;\lambda_{j}\;I_{j}^{2}\;\;\;\text{for
}j\in\\{S,B\\}\;,$ (4)

where $\lambda_{j}\in\mathbb{R}^{+}$ denotes the marginal cost parameter. It
is assumed that the parameters $w_{S}$, $w_{B}$, $\lambda_{S}$, $\lambda_{B}$,
and $b$ are known to the divisions and the headquarters. Incidentally, the
term, one half, is only for reasons of expediency. Please notice that the
supplying division’s net revenue generated by the internal quantity transfer
corresponds to the buying division’s transfer costs, i.e.,
$R_{S}=C_{B}=TP\cdot q$, and, hence, they do not appear in the headquarters’
profit $\Pi_{HQ}$. In of almost all transfer pricing models, the formulas are
represented by the transfer quantity $q$; the transfer price $TP$ does not
appear in any formula, but $TP$ can be calculated explicitly by using
substitution.

In a nutshell, the negotiated transfer pricing model investigated here
involves four dates (see Fig. 1): At date one, both divisions have to make an
investment decision independently of each other. Subsequently, supplying and
buying division independently observe the state variables $\theta_{S}$ and
$\theta_{B}$, respectively. At date three, the amount of intermediate products
is determined by the divisions and, finally, the headquarters’ profit as well
as the divisions’ profits are realized.

###  2.2 First-best solution

From the view of the headquarters, the decisions over investments and quantity
are taken in such a way that Eq. 3 is maximized. The solution of the two-stage
decision problem is received by backward induction starting with the quantity
decision on date three.666 The solutions presented here, as well as the
solutions of the most common transfer pricing methods, are described in
Wagner’s (2008) PhD thesis in more detail.

$q^{*}\in\underset{q\in\mathbb{R}^{+}}{\mathrm{arg\,max}}\;\Pi_{HQ}(q,\theta_{S},\theta_{B},I_{S},I_{B})$
(5)

Note, first-best solutions are indexed by a superscript ∗ and, in order to
guarantee that the first-best solutions are reached, divisions must be fully
rational at each stage of the decision problem, i.e., at each stage, the
action with the highest outcome must be selected. Since each division has
access to all relevant information, backward induction can be applied to
calculate the subgame perfect equilibrium.

On date three, the first order condition for maximizing the headquarters’
profit $\Pi_{HQ}$ with respect to $q$ is equal to

$\frac{\partial\Pi_{HQ}}{\partial q}=\theta_{B}-b\;q+I_{B}-\theta_{S}+I_{S}=0$
(6)

and, hence, the profit maximizing quantity is

$q^{*}=\frac{\theta_{B}-\theta_{S}+I_{S}+I_{B}}{b}\;.$ (7)

On date one, the expected headquarters’ profit $E[\Pi_{HQ}]$ is maximized with
respect to $I_{j}$, $j\in\\{S,B\\}$, because investment decisions are made
under uncertainty.

$I_{j}^{*}\in\underset{I_{j}\in\mathbb{R}^{+}}{\mathrm{arg\,max}}\;E[\Pi_{HQ}(q,\theta_{S},\theta_{B},I_{S},I_{B})]$
(8)

Differentiating the expected headquarters’ profit with respect to investments
yields

$I_{j}^{*}=\frac{E[q^{*}(\theta_{S},\theta_{B},I_{S},I_{B})]}{\lambda_{j}}\;.$
(9)

Now, substituting Eq. 7 into Eq. 8, one gets the following first-best expected
investments.

$\displaystyle I_{S}^{*}$ $\displaystyle=$
$\displaystyle\frac{\lambda_{S}\;E[\theta_{B}-\theta_{S}]}{b\;\lambda_{S}\;\lambda_{B}-\lambda_{S}-\lambda_{B}}$
(10) $\displaystyle I_{B}^{*}$ $\displaystyle=$
$\displaystyle\frac{\lambda_{B}\;E[\theta_{B}-\theta_{S}]}{b\;\lambda_{S}\;\lambda_{B}-\lambda_{S}-\lambda_{B}}$
(11)

Finally, substituting Eq. 10 and Eq. 11 into Eq. 7 implies the first-best
expected quantity.

$q^{*}=\frac{\theta_{B}-\theta_{S}}{b}+\frac{(\lambda_{S}+\lambda_{B})\;E[\theta_{B}-\theta_{S}]}{b\;(b\;\lambda_{S}\;\lambda_{B}-\lambda_{S}-\lambda_{B})}$
(12)

With the first-best solutions of the two-stage decision problem, the first-
best expected profit is given by the following expression.

$\Pi_{HQ}^{*}=\frac{1}{2}\;\frac{\lambda_{S}\;\lambda_{B}\;E[\theta_{B}-\theta_{S}]^{2}}{b\;\lambda_{S}\;\lambda_{B}-\lambda_{S}-\lambda_{B}}+\frac{Var[\theta_{B}-\theta_{S}]}{2b}$
(13)

The headquarters’ profit in Eq. 13 can be seen as a benchmark for the highest
feasible profit that can be achieved, if investments and quantity are set
equal to Eq. 10, Eq. 11, and Eq. 12, respectively.

###  2.3 Second-best solution

In negotiated transfer pricing models, both divisions simultaneously undertake
investments before the quantity negotiation takes place. Due to the fact that
each division is free to decide any amount of investment and quantity, the
headquarters has to provide an incentive compatibility for each division. In
order to ensure that both divisions report their state variables truthfully
and the negotiation leads to an ex post efficient quantity, the following
division incentives are commonly used.777 In general, incentives for efficient
investments do not necessarily provide incentives for an efficient quantity
and verse versa. For instance, in full-cost transfer pricing models, both
divisions are rewarded for efficient investments, but the negotiation does not
lead to an ex post efficient quantity (Baldenius et al. 1999).

$\displaystyle\Pi_{S}(q,\theta_{S},\theta_{B},I_{S},I_{B},\Gamma)$
$\displaystyle=$
$\displaystyle\Gamma\;M(q,\theta_{S},\theta_{B},I_{S},I_{B})-w_{S}(I_{S})$
(14) $\displaystyle\Pi_{B}(q,\theta_{S},\theta_{B},I_{S},I_{B},\Gamma)$
$\displaystyle=$
$\displaystyle(1-\Gamma)\;M(q,\theta_{S},\theta_{B},I_{S},I_{B})-w_{B}(I_{B})$
(15)

$\Pi_{S}$ and $\Pi_{B}$ denote the profit of the supplying division and the
profit of the buying division, respectively.
$M(q,\theta_{S},\theta_{B},I_{S},I_{B})$ stands for the headquarters’
contribution margin, i.e., $M=R_{B}-C_{S}$, and $\Gamma\in[0,1]$ represents
the share of the contribution margin achieved. In almost all negotiated
transfer pricing models, the negotiation process is modeled by a simple linear
surplus sharing rule, i.e., the supplying division receives a share $\Gamma$
and the buying division a fraction $(1-\Gamma)$ of the headquarters’
contribution margin. The parameter $\Gamma$ can be considered as the supplying
division’s bargaining power and, for $\Gamma=0.5$, the negotiation result is
equivalent to the Nash bargaining solution of the standard Rubinstein
bargaining game (e.g., Baldenius et al. 1999; Rubinstein 1982).

Again, the analysis of negotiated transfer pricing starts by backward
induction on date three. Since the headquarters delegates operation decisions
to its divisions, the supplying division and the buying division seek to
maximize their own divisional profits which are given by Eq. 14 and Eq. 15,
respectively. Starting with the quantity decision on date three, one gets

$q^{sb}=\frac{\theta_{B}-\theta_{S}+I_{S}+I_{B}}{b}\;.$ (16)

A closer look reveals that differentiating the divisional profits with respect
to $q$ leads to the same profit maximizing quantity given by Eq. 7, but the
first order conditions for maximizing the expected divisional profits
$E[\Pi_{j}]$, $j\in\\{S,B\\}$, with respect to $I_{j}$ lead to second-best
investments (labelled by the superscript $sb$).

$\displaystyle I_{S}^{sb}$ $\displaystyle=$
$\displaystyle\frac{\Gamma\;E[q^{sb}(\theta_{S},\theta_{B},I_{S},I_{B})]}{\lambda_{S}}$
(17) $\displaystyle I_{B}^{sb}$ $\displaystyle=$
$\displaystyle\frac{(1-\Gamma)\;E[q^{sb}(\theta_{S},\theta_{B},I_{S},I_{B})]}{\lambda_{B}}$
(18)

In the case of negotiated transfer pricing, the headquarters has to determine
how the contribution margin is shared between the divisions. Therefore, the
headquarters is faced with the following decision problem which has to be
solved before the decision-making process for the divisions begins.

$\Gamma^{sb}\in\underset{\Gamma\in[0,1]}{\mathrm{arg\,max}}\;E[\Pi_{HQ}(q,\theta_{S},\theta_{B},I_{S},I_{B},\Gamma)]$
(19)

Differentiating the expected headquarters’ profit with respect to $\Gamma$

$\frac{\partial}{\partial\Gamma}\Big{(}\big{(}E[\theta_{B}]-\frac{1}{2}bq^{sb}+I_{B}^{sb}\big{)}q^{sb}-\big{(}E[\theta_{S}]-I_{S}^{sb}\big{)}q^{sb}-\frac{1}{2}\lambda_{S}{I_{S}^{sb}}^{2}-\frac{1}{2}\lambda_{B}{I_{B}^{sb}}^{2}\Big{)}=0$
(20)

yields

$\Gamma^{sb}=\frac{b\;\lambda_{B}-1}{b\;(\lambda_{S}+\lambda_{B})-2}\;.$ (21)

By using substitution as in Sec. 2.2, one obtains the following second-best
solutions.

$\displaystyle I_{S}^{sb}$ $\displaystyle=$
$\displaystyle\frac{\lambda_{B}\;E[\theta_{B}-\theta_{S}]}{(b\;\lambda_{S}-1)\;(\lambda_{S}+\lambda_{B})}$
(22) $\displaystyle I_{B}^{sb}$ $\displaystyle=$
$\displaystyle\frac{\lambda_{S}\;E[\theta_{B}-\theta_{S}]}{(b\;\lambda_{B}-1)\;(\lambda_{S}+\lambda_{B})}$
(23) $\displaystyle q^{sb}$ $\displaystyle=$
$\displaystyle\frac{\lambda_{S}\;\lambda_{B}\;(b\;(\lambda_{S}+\lambda_{B})-2)\;E[\theta_{B}-\theta_{S}]}{(b\;\lambda_{S}-1)\;(b\;\lambda_{B}-1)\;(\lambda_{S}+\lambda_{B})}+\frac{\theta_{B}-\theta_{S}-E[\theta_{B}-\theta_{S}]}{b}$
(24) $\displaystyle\Pi_{HQ}^{sb}$ $\displaystyle=$
$\displaystyle\frac{1}{2}\;\frac{\lambda_{S}\;\lambda_{B}\;(b\;(\lambda_{S}+\lambda_{B})-1)\;E[\theta_{B}-\theta_{S}]^{2}}{(b\;\lambda_{S}-1)\;(b\;\lambda_{B}-1)\;(\lambda_{S}+\lambda_{B})}+\frac{Var[\theta_{B}-\theta_{S}]}{2b}$
(25)

According to Eq. 21, the headquarters assigns higher bargaining power to the
division that has a lower marginal cost parameter. If
$\lambda_{S}=\lambda_{B}$, then the surplus parameter is one half, i.e., both
divisions receive equal shares of the contribution margin achieved. Regardless
of the headquarters’ decision on $\Gamma$, Eq. 22 and Eq. 23 indicate that
both divisions are prone to underinvesting and, hence, the first-best quantity
resulting from the negotiation process cannot be reached either. As outlined
in the first-best case, the headquarters’ profit depends not only on
$\lambda_{j}$, $j\in\\{S,B\\}$, but also on the expected values of the state
variables and benefits, in particular, from the volatility of the markets.888
Please be aware that the variance of the difference between two stochastically
independent random variables is given by the sum of their variances. In
general it holds:
$Var[\theta_{B}\pm\theta_{S}]=Var[\theta_{B}]+Var[\theta_{S}]\pm
2Cov[\theta_{B},\theta_{S}]$.

##  3 An agent-based variant of negotiated transfer pricing with cognitively
bounded agents

###  3.1 Relaxed assumptions

Recall that the negotiated transfer pricing model (e.g., Edlin and
Reichelstein 1995; Fudenberg and Tirole 1991; Vaysman 1998) requires fully
individual rational utility maximizers. As mentioned in the introduction, such
assumptions are rather heroic. Therefore, this paper investigates negotiated
transfer pricing with divisions which (1) have no beliefs about the
rationality of other divisions nor how other divisions face their maximization
problem, (2) have no common knowledge of the opponent’s utility function, and
(3) are subject to cognitive limitations such as limits on the capacity of
human memory, restrictions on the speed of learning, and constraints in
foresight. In order to implement these relaxations of key assumptions, an
agent-based simulation is set up.

The negotiated transfer pricing model presented in the previous section
assumes that the divisions make decisions by means of their anticipated
information. Hence, the common knowledge assumption is required to determine
the optimal values of the decision variables $\Gamma$, $I_{j}$, and $q$,
$j\in\\{S,B\\}$. In the agentized version of the negotiated transfer pricing
model, the divisions know neither the profit function $\Pi_{j}$ nor the
expected value of the state variable $E[\theta_{j}]$ of the other division. In
addition, both divisions only share those parameters that are necessary for an
efficient quantity decision in their negotiation phase at date three. Table
LABEL:tab:Overview shows the parameters of negotiated transfer pricing
presented in Sec. 2 and summarizes the most important changes associated with
the agent-based variant of negotiated transfer pricing with cognitively
bounded agents. It should be emphasized once again that, in this paper,
cognitively bounded agents are modeled by fuzzy Q-learning agents, which are
introduced in Sec. 3.3.

###  3.2 What the agents have to learn

In the agent-based variant, both divisions have first to learn how their
environment works in which they are and, in order to do so, they have the
ability to store their individually learned information. For this purpose, the
divisions are able to process a manageable number of states (concretely, $25$
states are used in the simulation study to describe the agent’s memory; the
parameter settings for the simulations are discussed in Sec. 4 in more
detail). Moreover, the divisions have a learning rate which indicates how much
new information overrides old information and both divisions have a discount
factor which represents the agent’s foresight, i.e., how important future
rewards are to the agent. While the learning rate is fixed in the simulation
study, the discount factor is a scenario-based exogenous parameter which
varies in small steps. In addition to varying the discount factor, three
different exploration policies are examined, as the agents no longer
instantaneously see the consequences of their actions.

Recall that, in the organizational setting investigated here, the headquarters
determines the incentives for the divisions in such a way that the negotiation
over transfer price and quantity at date three leads to an ex post efficient
quantity (cf. Eq. 16). Consequently, the two divisions are only faced with the
investment decision at date one. Therefore, the agents have to learn which
amount of investment leads to which consequence. Since the simulation study
assumes that the agents have no prior knowledge about their consequences nor
about opponent’s decision-making behavior, the sequence of events within the
negotiated transfer pricing model is run through several times. In order to
indicate this additional time dimension, a time subscript $t\in\mathbb{N}$ for
the “inner loop” of the simulation is added to the model (a flow diagram of
the agent-based simulation is given in Sec. 4, Fig. 2).

###  3.3 The learning method

A common approach to describe individual learning behavior is reinforcement
learning (Sandholm and Crites 1996). This approach is applicable in cases
where the learning individuals (agents) are in an unknown environment and
choose actions for which they receive rewards, but they do not have the
ability to directly observe their opponents’ actions and, especially, they do
not have beliefs about the likely play of others (Feltovich 2000). In
reinforcement learning, a learning agent is able to perceive its environment,
can choose actions that can change its environmental state, and, of course,
the agent pursues a goal. Moreover, an agent uses its past experiences to find
an optimal policy to maximize its utility function.999 Note, a policy is a
mapping from environmental states to agent’s actions and determines the
learning agent’s way of behaving (Sutton and Barto 2018). In the following, a
brief introduction to Q-learning and its fuzzy version is provided.

####  3.3.1 Q-learning

Q-learning is a popular reinforcement learning approach to multi-agent
problems. The learning method by Watkins (1989) is based on estimating the
state-action value function (or Q-function) that ultimately gives the expected
utility of a given action in a given state. In Q-learning, an agent learns its
Q-values $\in\mathbb{R}$ by a simple iteration update which is given by

$Q_{t+1}[\boldsymbol{s}_{t},a_{t}]=(1-\alpha)\;Q_{t}[\boldsymbol{s}_{t},a_{t}]+\alpha\;\Big{(}r_{t}(\boldsymbol{s}_{t},a_{t})+\gamma\;\max\limits_{a\in\mathcal{A}}Q_{t}[\boldsymbol{s}_{t+1},a]\Big{)}\;,$
(26)

where
$\boldsymbol{s}_{t}=(s_{t,S},s_{t,B})\in\mathcal{S}\subset\mathbb{N}^{2},a_{t}\in\mathcal{A}\subset\mathbb{N}$,
and $r_{t}\in\mathbb{R}$ stand for state vector, action, and reward at time
$t\in\mathbb{N}$, respectively. For the sake of readability, the subscript
$j\in\\{S,B\\}$ is omitted from all variables in the formulas. In line with
the negotiated transfer pricing model in Sec. 2, action $a_{t}$ taken by the
supplying division (buying division) corresponds to the investment decision
$I_{S}$ ($I_{B}$), the state $s_{t,S}$ ($s_{t,B}$) represents the last action
$a_{t-1}$ chosen by the supplying division (buying division), and reward
$r_{t}$ that the supplying division (buying division) receives is equal to
$\Pi_{S}$ ($\Pi_{B}$).

$\mathcal{S}$ and $\mathcal{A}$ describe the state space and the action space,
respectively. Further, $\gamma\in[0,1)$ describes the discount factor and
$\alpha\in(0,1]$ is the learning rate.101010 Technical note: If the discount
factor is one, then the sum of the expected future rewards becomes infinite,
which implies that the Q-values keep growing as long as the simulation is
running. Hence, the Q-function does not converge. Q-learning can be seen as a
weighted average of old stored information and new received information from
the agent’s environment. Note that the values of the state-action value
function are stored in a lookup table (labelled with square brackets)
initialized with zero for all states and actions, i.e., there is no
information about the agent’s environment when the simulation is started.

####  3.3.2 Fuzzy Q-learning

In the conventional Q-learning approach, the Q-values are recorded in a look-
up table. If the number of states and actions increases, this method is highly
impractical and, especially, it is not applicable to cases of continuous
states or actions. Different authors have also reported poor performance due
to lack of convergence property in multi-agent settings (e.g., Agogino and
Tumer 2005; Georgila et al. 2014).

In this paper, an efficient adaptation of Watkins’ Q-learning method, called
fuzzy Q-learning, is applied to facilitate a continuous state space and action
space. Fuzzy Q-learning proposed by Glorennec (1994) is an universal
approximator of the Q-function defined in Eq. 26 and combines Q-learning and a
fuzzy rule-based system. The fuzzy rule-based system in this investigation has
the following form, which can be regarded as a zero order Takagi-Sugeno fuzzy
system.

$\displaystyle\text{fuzzy rule }i:\hskip 5.69054pt\text{IF }s_{t,S}\text{ IS
}L_{S}^{i}$ $\displaystyle\text{ AND }s_{t,B}\text{ IS }L_{B}^{i}\text{ THEN
}$ (27) $\displaystyle
Q_{t}(\boldsymbol{s}_{t},\boldsymbol{a}_{t})=\sum\limits_{i=1}^{N}\alpha_{i}(\boldsymbol{s}_{t})\;q_{t}[i,a_{t,i}]$
(28) $\displaystyle
A_{t}(\boldsymbol{s}_{t},\boldsymbol{a}_{t})=\sum\limits_{i=1}^{N}\alpha_{i}(\boldsymbol{s}_{t})\;a_{t}[i,a_{t,i}]$
(29)

The parameter $i\in\\{1,...,N\\}$ describes the index of fuzzy rules,
$N\in\mathbb{N}$ is the number of fuzzy rules,
$\boldsymbol{s}_{t}=(s_{t,S},s_{t,B})\in\mathcal{S}\subset\mathbb{R}^{2}$
represents the state vector,
$\boldsymbol{a}_{t}=(a_{t,1},...,a_{t,N})\in\\{1,...,K\\}^{N}$ denotes the
index vector of stored actions
$a_{t}[i,a_{t,i}]\in\mathcal{A}\subset\mathbb{R}$, $K\in\mathbb{N}$ is the
number of stored actions in each fuzzy rule,
$A_{t}(\boldsymbol{s}_{t},\boldsymbol{a}_{t})\in\mathbb{R}$ refers to the
inferred action, and $q_{t}[i,a_{t,i}]\in\mathbb{R}$ denotes the stored
q-values. As with the conventional Q-learning approach, the q-values are
stored in a look-up table, but fuzzy Q-learning requires much less memory
storage, since the table size depends on the number of fuzzy rules $N$ and on
the number of stored actions $K$ in each fuzzy rule. $N$ and $K$ rely on the
fuzzy partition of the state space $\mathcal{S}$ and on the discretization of
the action space $\mathcal{A}$, respectively. For the sake of simplicity, it
is assumed that each fuzzy rule has exactly $K$ possible actions.

Note, in fuzzy Q-learning, $\mathcal{S}$ and $\mathcal{A}$ are continuous
spaces and the inferred action $A_{t}(\boldsymbol{s}_{t},\boldsymbol{a}_{t})$
of the supplying division (buying division) corresponds to the investment
decision $I_{S}$ ($I_{B}$). Further, the fuzzy sets $L_{j}^{i}$ are usually
characterized by linguistic labels (e.g., “low”, “high”) and the function
$\alpha_{i}(\boldsymbol{s}_{t})$ denotes the truth value of rule $i$ given the
state vector $\boldsymbol{s}_{t}$. Finally, Q-values
$Q_{t}(\boldsymbol{s}_{t},\boldsymbol{a}_{t})$ and actions
$A_{t}(\boldsymbol{s}_{t},\boldsymbol{a}_{t})$ are inferred from Eq. 28 and
Eq. 29, respectively.

In fuzzy Q-learning, the Q-function is no longer a look-up table, since the
Q-values are estimated by a linear parameterized approximation of a few stored
q-values, where the “weights” $\alpha_{i}(\boldsymbol{s}_{t})$ are commonly
generated by the T-norm product

$\alpha_{i}(\boldsymbol{s}_{t})=\mu_{L_{S}^{i}}(s_{t,S})\cdot\mu_{L_{B}^{i}}(s_{t,B})\;,$
(30)

where $\mu_{L_{j}^{i}}(s_{t,j})\in[0,1]$ denotes the membership function of
rule $i$ in state $s_{t,j}$, $j\in\\{S,B\\}$. The T-norm (or triangular norm,
see, e.g., Zimmermann (2011)) is a type of binary operation that is often used
in fuzzy logic to model the AND operator in Eq. 27. Usually, the membership
function (or membership grade) is defined on the interval $[0,1]$, which
implies that if an object has a membership grade of one (zero) in a fuzzy set,
then the object is absolutely (not) in that fuzzy set. In addition, the
membership functions considered here are set in such a way that the strong
fuzzy partition is fulfilled, i.e.,
$\sum_{i=1}^{N}\alpha_{i}(\boldsymbol{s}_{t})=1$ for each
$\boldsymbol{s}_{t}\in\mathcal{S}$.

Similar to the conventional Q-learning method, the q-values are updated by

$q_{t+1}[i,a_{t,i}]=q_{t}[i,a_{t,i}]+\alpha_{i}(\boldsymbol{s}_{t})\;\Delta
Q_{t}(\boldsymbol{s}_{t},\boldsymbol{a}_{t})\;,$ (31)

where $\Delta Q_{t}(\boldsymbol{s}_{t},\boldsymbol{a}_{t})$ is the temporal
difference error which is given by

$\alpha\;\Big{(}r_{t}(\boldsymbol{s}_{t},A_{t}(\boldsymbol{s}_{t},\boldsymbol{a}_{t}))\;+\;\gamma\;\sum\limits_{i=1}^{N}\alpha_{i}(\boldsymbol{s}_{t+1})\max\limits_{k\in\\{1,...,K\\}}q_{t}[i,k]\;-\;Q_{t}(\boldsymbol{s}_{t},\boldsymbol{a}_{t})\Big{)}\;.$
(32)

According to Eq. 31, up to $N$ q-values per time step can be adjusted and,
therefore, the Q-function converges more quickly in fuzzy Q-learning.111111
Roughly speaking, the Q-function converges to its optimum, when a sufficient
number of time steps has been performed and the stored q-values change only
slightly, or if the temporal difference error $\Delta
Q_{t}(\boldsymbol{s}_{t},\boldsymbol{a}_{t})$ goes to zero, i.e., $\Delta
Q_{t}(\boldsymbol{s}_{t},\boldsymbol{a}_{t})\to 0$ for $t\to\infty$. For the
precise Q-learning conditions which ensure that the Q-function converges (from
any initial state) see, e.g., Watkins and Dayan (1992). According to the
parameter settings in Sec. 4, up to $4$ q-values are updated simultaneously
per time step. Moreover, fuzzy Q-learning is simple, has low computational
demands per iteration, and, with relatively few environmental interactions,
good results can be achieved. Nevertheless, there are more sophisticated
learning techniques in reinforcement learning (for an overview see, e.g.,
Sutton and Barto (2018)).

###  3.4 Exploration policy

A very key part in reinforcement learning is the trade-off between exploration
and exploitation. In order to select “good” actions, an agent has to explore
its environment and, in particular, has to learn which action provides which
reward. As with other reinforcement learning approaches, a balance between
exploration and exploitation has to be found. Therefore, an agent is
confronted with the trade-off between choosing the currently optimal action
(exploitation) and choosing a varied action with the prospect of a higher
reward in the future (exploration) (Sutton and Barto 2018). Nevertheless, the
agent needs the possibility to explore all states and all actions frequently
enough to ensure the convergence of the Q-function to an optimum. In the
following, three exploration policies, which are applied in the simulation
study, are explained.

####  3.4.1 Boltzmann exploration policy

A commonly used and very sophisticated exploration policy is the Boltzmann
exploration policy (Cesa-Bianchi et al. 2017).121212 For other exploration
policies see, e.g., Thrun (1992), and, by the way, the Boltzmann exploration
policy is sometimes referred to as Gibbs or softmax exploration policy (Sutton
and Barto 2018). For each action $a_{t}[i,k]$, $i\in\\{1,...,N\\}$ and
$k\in\\{1,...,K\\}$, the Boltzmann probability mass function is given by

$P_{t,i,k}(a_{t}[i,k])=\frac{exp\big{(}{q_{t}[i,k]/\beta(t)}\big{)}}{\sum_{l=1}^{K}exp\big{(}{q_{t}[i,l]/\beta(t)}\big{)}}\;,$
(33)

where $\beta(t)\in\mathbb{R}^{+}$ denotes the experimentation tendency
(Waltman and Kaymak 2008). According to Eq. 33, it holds
$\sum_{k=1}^{K}P_{t,i,k}=1$ for each fuzzy rule $i$ and time step $t$. The
Boltzmann distribution states that actions with higher q-values are more
likely to be selected than actions with lower q-values. The degree of
randomness for exploration is controlled by the experimentation tendency
(Powell and Ryzhov 2012) and, therefore, $\beta(t)$ should change slowly over
time. In particular, if $\beta(t)$ tends to infinity, the shape of the
Boltzmann distribution converges to the discrete uniform distribution, which
means that all actions have the same probability to being chosen (pure
exploration). Contrary, if $\beta(t)$ decreases toward zero, only the action
with the highest Q-value is selected (pure exploitation).

For the long-term behavior of self-learning agents, the experimentation
tendency should slowly decrease over time in order to reduce exploration
(Dearden et al. 1998). Hence, the experimentation tendency is calculated by a
simple rational function

$\beta(t)=\frac{\beta_{1}}{\beta_{2}+t}\;,$ (34)

whereby the experimentation tendency parameters
$(\beta_{1},\beta_{2})\in\mathbb{R}^{2}$ should be set in such a way that the
learning agents have a long time to explore for good policies, but also time
to exploit them.131313 Rational functions, like Eq. 34, have the advantage
over linear functions that the experimentation tendency decreases faster at
the beginning of a simulation run, but it never reaches zero. Technical note:
the experimentation tendency $\beta(t)$ should not become too small during a
simulation run, otherwise the expression $exp(q_{t}[i,k]/\beta(t))$ will be
too large and this could endanger the numerical stability of the simulation.

Note, after all probabilities have been calculated according to Eq. 33, the
index of each stored action is drawn from

$a_{t,i}\thicksim\bigg{(}{1\atop P_{t,i,1}},...,{K\atop P_{t,i,K}}\bigg{)}$
(35)

for each fuzzy rule $i$.

####  3.4.2 $\epsilon$-greedy exploration policy

Another common and fairly easy exploration policy is the so-called
$\epsilon$-greedy exploration policy, which means that with probability
$1-\epsilon(t)$ the action with the highest q-value is chosen, while with
probability $\epsilon(t)\in[0,1]$ a random action is selected (Sutton and
Barto 2018). For each fuzzy rule $i\in\\{1,...,N\\}$, the index of the stored
action is calculated by the following rule:

$a_{t,i}=\begin{cases}\vspace{3mm}\underset{k\in\\{1,...,K\\}}{\mathrm{arg\,max}}q_{t}[i,k]&\text{with
probability $1-\epsilon(t)$}\\\ \thicksim
Unif\Big{(}\\{1,...,K\\}\Big{)}&\text{with probability
$\epsilon(t)$}\end{cases}$ (36)

In the simulation study, $\epsilon(t)$ is a decreasing linear function of the
time

$\epsilon(t)=\begin{cases}\vspace{3mm}\epsilon_{1}-\epsilon_{2}\;t&\text{for
$t\leq T_{L}$}\\\ 0&\text{for $t>T_{L}$}\end{cases}$ (37)

with the two properties that, at the beginning of a simulation run,
$\epsilon(1)$ is one, which indicates that the action-selection is total
random (pure exploration) and, in the end, $\epsilon(T_{L})$ is zero (pure
exploitation). The parameter $T_{L}$ is discussed in Sec. 4.

####  3.4.3 Upper confidence bound exploration policy

For the third and last exploration policy, the so-called upper confidence
bound exploration policy (often abbreviated as the UCB policy) is adapted,
which was also proposed for multi-armed bandit problems (Auer et al. 2002).
The idea behind this exploration policy is that not only the current q-values
are used for the action-selection, but also the uncertainty that comes along
with that selection. One way of doing this is to select an action according to
the following rule (Sutton and Barto 2018):

$a_{t,i}=\begin{cases}\vspace{3mm}\underset{k\in\\{1,...,K\\}}{\mathrm{arg\,max}}q_{t}[i,k]+c_{1}\sqrt{\frac{ln(t)}{N_{t}[i,k]}}&\text{if
$\forall N_{t}[i,k]>0$}\\\ \thicksim
Unif\Big{(}\\{k\in\\{1,...,K\\}\;|\;N_{t}[i,k]=0\\}\Big{)}&\text{else}\end{cases}$
(38)

The parameter $c_{1}\in\mathbb{R}$ describes the degree of exploration and
$N_{t}[i,k]\in\mathbb{N}$ is the number of times that the index of the stored
action $k$ in fuzzy rule $i$ has been chosen prior to time $t$. At the
beginning of a simulation run, all $N_{t}[i,k]$ are initialized with zero and,
each time $a_{t,i}$ is determined, $N_{t}[i,a_{t,i}]$ is incremented by one,
and, hence, the “uncertainty term” is reduced.

Note that actions that have never been chosen are given preference over
actions that have already been chosen and the use of the natural logarithm
implies that the uncertainty increases slightly over time. If $c_{1}$ is set
too high, then the distribution of the UCB policy converges to the discrete
uniform distribution. If $c_{1}$ is set to zero, then the highest q-value
determines which action is chosen.

##  4 Parameter settings and simulation setup

The agent-based simulation is conducted in four main steps: First, the
scenario in which both divisions have the same marginal cost parameter is
studied, i.e., $\lambda_{S}=\lambda_{B}$ and, according to Eq. 21, this yields
to a surplus sharing parameter $\Gamma$ of $0.5$, which means that each
division receives the same fraction of the contribution margin achieved. In
the next step, the case where the divisional investment costs are not
symmetrical is examined, which means that the contribution margin is not
divided symmetrically either. A third analysis studies whether the divisions
make higher investment decisions, if the surplus sharing parameter $\Gamma$
has different values than the theory of subgame perfection equilibrium
recommends. In the first three investigations, the standard deviations of the
state variables is set to zero, which implies that there are no stochastic
fluctuations that affect the decision-making process, and, in addition, it is
supposed that the divisions explore their environment using the Boltzmann
exploration policy. In the fourth and final step, a sensitivity analysis with
regard to the standard deviations of the state variables and the exploration
policies is carried out in order to check whether the observed results also
occur in stochastic environments with different exploration settings.141414 A
simulation study should include a systematic sensitivity analysis to verify
the stability of the simulation outcomes (e.g. Davis et al. 2007; Guerrero and
Axtell 2011).  For a better understanding, the parameter settings are
successively explained in more detail and the procedure of the agent-based
simulation is illustrated (the exogenous parameters are summarized in Table 1,
while the procedure of the agent-based simulation is depicted in Fig. 2).

Table 1: Parameter settings Fixed exogenous parameters | Values
---|---
Number of simulation runs | $10,000$
Number of time steps per simulation run | $T=1,100$
Time steps to learn the Q-function | $T_{L}=1,000$
Time steps to evaluate the outcome | $T_{E}=100$
Slope of inverse demand function | $b=12$
Expected values of the state variables |  $E[\theta_{S}]=60$ and $E[\theta_{B}]=100$
Action space | $\mathcal{A}=\\{0,5,...,50\\}$
State space | $\mathcal{S}\hskip 1.42262pt=\\{0,12.5,...,50\\}\times\\{0,12.5,...,50\\}$
Number of fuzzy rules | $N=25$
Number of stored actions in each fuzzy rule | $K=11$
Learning rate | $\alpha=0.5$
Scenario-based exogenous parameters | Values
Marginal cost parameter | $\lambda_{S}\in\\{1/2,\hskip 0.85358pt7/12,\hskip 0.85358pt2/3,\hskip 0.85358pt3/4,\hskip 0.85358pt5/6\\}$
|  $\approx\\{0.5,0.58,0.67,0.75,0.83\\}$
| $\lambda_{B}=1-\lambda_{S}$
Supplying division’s surplus sharing parameter | $\Gamma\in\\{0.1,0.2,...,0.9\\}$
Discount factor | $\gamma\in\\{0.0,0.1,...,0.9\\}$
Standard deviations of the state variables | $\sigma[\theta_{S}]=\sigma[\theta_{B}]\in\\{0,5,10\\}$
Exploration policies: |
Boltzmann exploration policy |  $\beta_{1}=12487.5$ and $\beta_{2}=248.75$
$\epsilon$-greedy exploration policy |  $\epsilon_{1}=1+1/999$ and $\epsilon_{2}=-1/999$
Upper confidence bound exploration policy | $c_{1}=30$
Figure 2: Flow diagram of the agent-based simulation using the Boltzmann
exploration policy.

This simulation study focuses on settings in which the learning behavior of
agents is modeled by fuzzy Q-learning. It is assumed that each division has an
individual constant marginal cost parameter $\lambda_{j}$, $j\in\\{S,B\\}$.
Since there are infinitely many combinations that can be examined, the
simulation experiments only focus on the following convex combination, namely,
$\lambda_{S}+\lambda_{B}=1$ with $\lambda_{j}\geq 0$. In order to analyze
scenarios with surplus sharing parameters $\Gamma$ ranging from $0.1$ to $0.9$
in steps of $0.1$ (and in order that the first-best and the second-best case
have integer solutions for $\Gamma=0.5$), the slope of the inverse demand
function $b$ is set to $12$ as well as the expected values of the state
variables $\theta_{S}$ and $\theta_{B}$ is set to $60$ and $100$,
respectively. The corresponding $\lambda_{j}$ values are given in Table 1,
whereas the solutions of subgame perfect equilibrium are listed in Table 2. By
the way, scenarios where $\Gamma$ is one (zero) lead, according to Eq. 9 and
Eq. 17, to the result that the supplying division (buying division) chooses
first-best investments, but the other division has no incentive to invest as
any investment cause divisional investment costs and, at the same time, it
does not benefit from the resulting profit. Hence, such scenarios are less
interesting to analyze and, due to the fact that both divisions have the same
quadratic cost structure (cf. Eq. 4) and that $\lambda_{B}=1-\lambda_{S}$, it
is sufficient to study scenarios for $\lambda_{S}$ ranging from $0.5$ to
$0.83$.

Table 2: First-best and second-best solutions resulting from the concept of
subgame perfect equilibrium for $b=12$, $E[\theta_{S}]=60$, and
$E[\theta_{B}]=100$. First-best solutions
---
$\Gamma$ | $\lambda_{S}$ | $\lambda_{B}$ | $I_{S}^{*}$ | $I_{B}^{*}$ | $q^{*}$ | $\Pi_{S}^{*}$ | $\Pi_{B}^{*}$ | $\Pi_{HQ}^{*}$
$0.1$ | $0.83$ | $0.17$ | $10$ | $50$ | $8.33$ | $0$ | $166.67$ | $166.67$
$0.2$ | $0.75$ | $0.25$ | $8$ | $24$ | $6$ | $19.2$ | $100.8$ | $120$
$0.3$ | $0.67$ | $0.33$ | $8$ | $16$ | $5.33$ | $29.87$ | $76.8$ | $106.67$
$0.4$ | $0.58$ | $0.42$ | $8.70$ | $12.17$ | $5.07$ | $39.70$ | $61.75$ | $101.45$
$0.5$ | $0.5$ | $0.5$ | $10$ | $10$ | $5$ | $50$ | $50$ | $100$
$0.6$ | $0.42$ | $0.58$ | $12.17$ | $8.70$ | $5.07$ | $61.75$ | $39.70$ | $101.45$
$0.7$ | $0.33$ | $0.67$ | $16$ | $8$ | $5.33$ | $76.8$ | $29.87$ | $106.67$
$0.8$ | $0.25$ | $0.75$ | $24$ | $8$ | $6$ | $100.8$ | $19.2$ | $120$
$0.9$ | $0.17$ | $0.83$ | $50$ | $10$ | $8.33$ | $166.67$ | $0$ | $166.67$
Second-best solutions
$\Gamma$ | $\lambda_{S}$ | $\lambda_{B}$ | $I_{S}^{sb}$ | $I_{B}^{sb}$ | $q^{sb}$ | $\Pi_{S}^{sb}$ | $\Pi_{B}^{sb}$ | $\Pi_{HQ}^{sb}$
$0.1$ | $0.83$ | $0.17$ | $0.74$ | $33.33$ | $6.17$ | $22.63$ | $113.17$ | $135.80$
$0.2$ | $0.75$ | $0.25$ | $1.25$ | $15$ | $4.69$ | $25.78$ | $77.34$ | $103.13$
$0.3$ | $0.67$ | $0.33$ | $1.90$ | $8.89$ | $4.23$ | $31.04$ | $62.08$ | $93.12$
$0.4$ | $0.58$ | $0.42$ | $2.78$ | $5.83$ | $4.05$ | $37.13$ | $51.99$ | $89.12$
$0.5$ | $0.5$ | $0.5$ | $4$ | $4$ | $4$ | $44$ | $44$ | $88$
$0.6$ | $0.42$ | $0.58$ | $5.83$ | $2.78$ | $4.05$ | $51.99$ | $37.13$ | $89.12$
$0.7$ | $0.33$ | $0.67$ | $8.89$ | $1.90$ | $4.23$ | $62.08$ | $31.04$ | $93.12$
$0.8$ | $0.25$ | $0.75$ | $15$ | $1.25$ | $4.69$ | $77.34$ | $25.78$ | $103.13$
$0.9$ | $0.17$ | $0.83$ | $33.33$ | $0.74$ | $6.17$ | $113.17$ | $22.63$ | $135.80$

Next, the choice of the action space $\mathcal{A}$ is discussed. If the number
of possible investment decisions is too small, the learning behavior of both
divisions cannot be thoroughly investigated, and, on the other hand, if the
size of the action space is chosen too large, then learning is slowed. In
order to ensure that the first-best investment decisions are in the action
space in all examined scenarios, the action space must have at least a minimum
size of $50$. Since it can be guessed that a human decision-maker can only
differentiate between a limited number of actions, the stored actions
$a_{t}[i,k]\in\mathcal{A}$ vary between $0$ and $50$ in steps of $5$.
Consequently, there are $11$ stored actions for each fuzzy rule
$i\in\\{1,...,N\\}$. Recall that, according to Eq. 29, regardless of whether
the stored actions have integer values or not, the inferred action $A_{t}$
(which corresponds to the agent’s investment decision) is a real number.

Also for the state space $\mathcal{S}$, it is supposed that the number of
fuzzy rules should be small. In order to deal with $5$ membership functions in
each state space dimension, the values of the state space range from $0$ to
$50$ by a step size of $12.5$; thus, there are $25$ fuzzy rules in total.
According to Zimmermann’s (2011) suggestions, triangular membership functions
are applied (cf. Fig. 3) because they are simple, have low computing power per
iteration, and are easy to interpret. For example, if the state of the
supplying division $s_{t,S}$ is $10$, then the membership grade
$\mu_{L_{S}^{1}}$ of fuzzy rule $1$ is $0.2$, whereas the membership grade
$\mu_{L_{S}^{2}}$ of fuzzy rule $2$ is $0.8$, and all other membership grades
$\mu_{L_{S}^{3}}$, $\mu_{L_{S}^{4}}$, and $\mu_{L_{S}^{5}}$ are zero, or in
other words, the supplying division is in the “very low investment range” with
a fraction of $20\%$ and in the “low investment range” with a share of $80\%$.
Incidentally, increasing the number of fuzzy rules in the parameter settings
just means that the agents need more time to learn their environment; their
decision-making behavior can be exhibited quite well with only $5$ membership
functions. Note, according to the triangular membership functions in Fig. 3,
up to $4$ q-values can be updated simultaneously per time step ($2$ raised to
the power of $dim(\mathcal{S})$).

Figure 3: The fuzzy sets considered here consist of $5$ triangular membership
functions in each state space dimension $j\in\\{S,B\\}$, which can be
characterized by linguistic labels ranging from “Very low” to “Very high”.

Furthermore, the research is limited to just one learning rate, namely,
$\alpha=0.5$, because it can be assumed that, for long-term decisions, old
stored information is just as important as new received information.
Regardless of the selection of the learning rate, the Q-function converges
(sometimes faster and sometimes slower) to an optimum with the parameter
settings giving in Table 1. Note, if the learning rate is set too high, the
learned information will be regularly overwritten, which is usually not
recommended as this requires more time steps for convergence and, in
particular, stochastic fluctuations during learning make convergence
difficult. On the other hand, if the learning rate is chosen too small, many
more time steps are required for convergence.

In the agent-based simulation, the number of time steps to learn the
Q-function $T_{L}\in\mathbb{N}$ is mainly based on the cardinalities of
$\mathcal{A}$ and $\mathcal{S}$; other factors like $\alpha$, the degree of
exploration, and the extent of stochastic fluctuations also influence the
learning time, but to take them into account explicitly would be too time-
consuming. Since each agent has $275$
($=|\mathcal{A}|\cdot|\mathcal{S}|^{2}=11\cdot 5^{2}$) different q-values and
the maximum number of q-values that are updated per time step is $4$, the
number of time steps to learn the Q-function is set to $1,000$ so that a
sufficient number of state-action pairs are visited. The pre-generated
simulations also suggest that a learning time of $1,000$ is sufficient to
guarantee convergence in all examined scenarios. After determining the number
of time steps required for learning the Q-function, further $100$ time steps
are simulated to evaluate the agents’ outputs, hence $T_{E}=100$ and,
consequently, the number of time steps per simulation run $T$ is set to
$1,100$.151515 Please be aware that even if the Q-function converges, the
stochastic fluctuations of the environment affect the divisions’ profits. In
order to take the volatility of the markets into account, an observation
period of $100$ time steps is chosen to evaluate the simulation outcomes.

It should be emphasized once again that this study aims to analyze negotiated
transfer pricing with cognitively bounded agents. This means that not only the
number of actions or the number of fuzzy rules is limited, but also the
possibility of observing all state-action pairs infinitely often. Therefore,
the simulation ends after $1,100$ time steps, even if not all state-action
pairs have been visited.

Besides the surplus sharing parameter $\Gamma$, also the discount factor
$\gamma$ varies from $0$ to $0.9$ in steps of $0.1$. The discount factor is
set exogenously at the very beginning of a simulation run and represents the
agent’s foresight. A discount factor of zero indicates that the agent is very
myopic, which implies that the agent cannot observe future rewards.
Conversely, if $\gamma$ goes to one, the agent becomes more non-myopic, so the
agent will place more emphasis on future rewards. Recall, when $\gamma$ is set
to one, then the Q-function does not converge.

Finally, the state variables $\theta_{S}$ and $\theta_{B}$ are captured in a
normally distributed random variable with mean $60$ and $100$, respectively.
To describe the turbulence of the environment in which the agents operate, the
standard deviations of the state variables are set to $0$, $5$, and $10$,
which can be interpreted as a deterministic environment, an environment with
minor stochastic fluctuations, and an environment with considerable volatility
on the markets.

In addition to varying the state variables, the sensitivity of the results is
checked in regard to three different exploration policies. Inspired by the
work of Tijsma et al. (2016), this paper focuses on the Boltzmann,
$\epsilon$-greedy, and upper confidence bound exploration policy. For the
introduced Boltzmann exploration policy in Sec. 3.4.1, one has to define the
exploration tendency parameters $\beta_{1}$ and $\beta_{2}$ in Eq. 34. The
pre-generated simulations reveal that an experimentation tendency $\beta(t)$
of around $50$ at the beginning of a simulation run is a good choice because
the agents are able to explore many action-state pairs. On the other hand, a
value of $10$ at the end of learning is sufficient for the convergence of the
Q-function and, from a technical point of view, $\beta(t)$ ranges in a value
domain in which no numerical instabilities occur during a simulation run.
Solving Eq. 34 with the two associated boundary conditions ($\beta(1)=50$, and
$\beta(T_{L})=10$) leads to the Boltzmann exploration tendency parameters
given in Table 1. Setting the exploration parameters $\epsilon_{1}$ and
$\epsilon_{2}$ for the $\epsilon$-greedy exploration policy is simple because
$\epsilon(1)=1$ and $\epsilon(T_{L})=0$. The associated $\epsilon$-greedy
exploration parameters are reported in Table 1. Finally, the UCB exploration
parameter $c_{1}$ has to specify. According to the pre-generated simulations,
a value of about $30$ is a good choice for $c_{1}$. Nevertheless, balancing
the parameters of an exploration policy is one of the most challenging tasks
in reinforcement learning (Tijsma et al. 2016).

Since the first state of each fuzzy Q-learning agent is determined by a
uniform random number generator and the choice of action is based on
stochastic exploration policies, each scenario is carried out $10,000$ times
and, due to the coefficient of variation (the ratio of the standard deviation
to the mean), $10,000$ simulation runs are sufficient to express the precision
and repeatability of the simulation study.

##  5 Results and discussion

This section analyzes the results of the agent-based simulation in four main
steps: (1) the results for symmetric marginal cost parameters is presented,
i.e., $\lambda_{S}=\lambda_{B}=0.5$ and, consequently, $\Gamma=0.5$. (2) the
scenario in which the marginal cost parameters are not symmetrical is
examined. For this purpose, the case where $\lambda_{S}$ is $0.83$ is
selected, which implies a $\Gamma$ of $0.1$. (3) different surplus sharing
parameters $\Gamma$ are tested to see what effect they have on the divisions’
investment decisions and the headquarters’ profit and (4) an extensive
sensitivity analysis is performed to verify whether the results are robust to
changes in volatility of the markets and exploration policy. An overview of
the investigated scenarios is given in Table 3.

Table 3: Scenario overview. | Seller’s mar- | Seller’s sur- | Discount | Standard |
---|---|---|---|---|---
Scenario | ginal cost | plus sharing | factor | deviation | Exploration policy
| $\lambda_{S}$ | $\Gamma$ | $\gamma$ | $\sigma[\theta_{S}]$ |
$1$ | $0.5$ | $0.5$ | $(0,0.9)$ | $0$ | Boltzmann
$2$ | $0.83$ | $0.1$ | $(0,0.9)$ | $0$ | Boltzmann
$3$ | $(0.5,0.83)$ | $(0.1,...,0.9)$ | $(0,...,0.9)$ | $0$ | Boltzmann
$4$ | $(0.5,...,0.83)$ | $(0.1,...,0.9)$ | $(0,...,0.9)$ | $(0,5,10)$ | (BM, Greedy, UCB)

###  5.1 Results for symmetric marginal cost parameters

In a first step, the scenario where each division has the same marginal cost
parameter is investigated. According to Eq. 21, the headquarters assigns equal
bargaining power to both divisions. The solutions resulting from the concept
of subgame perfection equilibrium are reported in Table 2. Recall that the
first-best solution can be considered as the best-case scenario in which both
divisions make optimal decisions. In addition, the standard deviations of the
state variables are set to zero so that the agents’ decision-making process is
not influenced by stochastic environmental fluctuations and, further, the
Boltzmann exploration policy is applied for the action-selection of
investments.

In order to get a better understanding of the decision-making behavior of
fuzzy Q-learning agents, the simulation study starts with the case of
divisions’ investment decisions with different levels of foresight. Recall,
the agent’s foresight is represented by the discount factor $\gamma$ in the
fuzzy Q-learning approach. A discount factor of zero indicates that the agent
is absolutely myopic and, therefore, does not take any future reward into
account. On the other hand, a discount factor close to one implies that the
agent strives for a long-term high reward.

Figure 4: Modified boxplots of seller’s investment $I_{S}$, buyer’s investment
$I_{B}$, and headquarters’ profit $\Pi_{HQ}$ for myopic fuzzy Q-learning
agents in the scenario $1$, i.e., $\lambda_{S}=\lambda_{B}=0.5$, $\Gamma=0.5$,
$\gamma=0$, $\sigma[\theta_{S}]=\sigma[\theta_{B}]=0$, and exploration policy
is Boltzmann.

Figure 5: Modified boxplots of seller’s investment $I_{S}$, buyer’s investment
$I_{B}$, and headquarters’ profit $\Pi_{HQ}$ for non-myopic fuzzy Q-learning
agents in the scenario $1$, i.e., $\lambda_{S}=\lambda_{B}=0.5$, $\Gamma=0.5$,
$\gamma=0.9$, $\sigma[\theta_{S}]=\sigma[\theta_{B}]=0$, and exploration
policy is Boltzmann.

Fig. 4 and 5 depict the divisions’ investment decisions and the related
headquarters’ profit for myopic and non-myopic fuzzy Q-learning agents,
respectively. While myopic agents invest only about as much as in the classic
hold-up problem, non-myopic agents invest optimally. Since there is no
stochastic influence in this scenario setting, the headquarters’ profit
depends solely on the divisions’ investment decisions. A closer analysis
reveals that the investment decisions of myopic (non-myopic) agents, averaged
over the last $100$ selected investments of $10,000$ simulation runs, are
normally distributed with mean $5.04$ ($9.59$) and standard deviation $0.43$
($3.17$); compared to the second-best solution, fully individual rational
utility maximizers only invest an amount of $4$. Further for myopic (non-
myopic) agents, the headquarters’ profit is almost normally distributed with
mean $87.9$ ($93.48$) and standard deviation $1.31$ ($5.14$) and, especially,
has a slightly negative skewness of about $-0.29$ ($-1.41$). The findings
suggest that, not only from the view of the headquarters, it is better when
the divisions act with a high degree of foresight, it is also better for both
divisions to seek for high future rewards. Ultimately, it is natural for
competing divisions to take expected future rewards into account to maximize
their own profits.

Be aware that the thick black line in the modified boxplots represents the
median of $10,000$ observations per time step and, due to the fact that the
mean value is not robust against outliers, the mean value of the headquarters’
profit is below the median value in case of a negative skewness. Additionally,
note that, in the modified boxplots used here, the distance between the 25th
and 75th percentile is visualized in gray, the thin black lines denote the
whiskers, and, for improved readability, outliers beyond the wiskers are not
depicted.

###  5.2 Results for non-symmetric marginal cost parameters

In the next step, the case where the divisions’ marginal cost parameters are
not symmetrical is analyzed. According to the selected convex combination,
$\lambda_{S}+\lambda_{B}=1$, there are four possible scenarios with non-
symmetric marginal cost parameters, of which the scenario with
$\lambda_{S}=0.83$ is examined. In the case that the seller’s marginal cost
parameter is greater than that of the buyer, the headquarters assigns the
seller a lower bargaining power (cf. Eq. 21). For myopic agents, this
recommended asymmetrical distribution of bargaining power leads to investments
in the area of the second-best solution (cf. Fig. 6). In the case of non-
myopic agents (cf. Fig. 7), the supplying division also invest in the area of
the second-best solution, but the buying division makes investment decisions
that are even significantly below the second-best solution. In detail, the
simulation results show that, for $\gamma=0$, the seller’s (buyer’s)
investments are normally distributed with mean $1.93$ ($32.85$) and standard
deviation $0.52$ ($3.47$). For $\gamma=0.9$, the seller’s investments are
approximately gamma distributed with shape parameter $1.10$ and scale
parameter $1.29$, whereas the buyer’s investments are approximately normally
distributed with mean $26.94$ and standard deviation $8.54$. Additionally, the
headquarters’ profit resulting from myopic divisions is normally distributed
with mean $136.87$ and standard deviation $4.21$, while, for non-myopic
divisions, $\Pi_{HQ}$ is approximately normally distributed with mean $127.51$
and standard deviation $13.43$. According to the simulation outputs, the
headquarters achieves higher profits with myopic agents than with non-myopic
agents.

Figure 6: Modified boxplots of seller’s investment $I_{S}$, buyer’s investment
$I_{B}$, and headquarters’ profit $\Pi_{HQ}$ for myopic fuzzy Q-learning
agents in the scenario $2$, i.e., $\lambda_{S}=0.83$, $\lambda_{B}=0.17$,
$\Gamma=0.1$, $\gamma=0$, $\sigma[\theta_{S}]=\sigma[\theta_{B}]=0$, and
exploration policy is Boltzmann.

Figure 7: Modified boxplots of seller’s investment $I_{S}$, buyer’s investment
$I_{B}$, and headquarters’ profit $\Pi_{HQ}$ for non-myopic fuzzy Q-learning
agents in the scenario $2$, i.e., $\lambda_{S}=0.83$, $\lambda_{B}=0.17$,
$\Gamma=0.1$, $\gamma=0.9$, $\sigma[\theta_{S}]=\sigma[\theta_{B}]=0$, and
exploration policy is Boltzmann.

The results for non-symmetrical marginal cost parameters indicate that
divisions that have a high discount factor still invest much less than
divisions that exhibit a low discount factor. However, the resulting question
is what causes this negative effect, as foresight is usually a positive
learning characteristic. To gain some insights on the effects of $\gamma$
given $\Gamma$, additional scenarios are considered in which the discount
factor $\gamma$ is fixed and the surplus sharing parameter $\Gamma$ is varied
in small steps.

###  5.3 Effects of different surplus sharing parameters

In order to figure out what effects do other $\Gamma$ constellations have on
the hold-up problem, further simulations are conducted by varying the surplus
sharing parameter in small steps. In addition, the value range for the
discount factor is expanded to cover a broader spectrum of the agent’s
foresight. The simulation outputs are summarized in Fig. 8 and Fig. 9 for
$\lambda_{S}=0.5$ and $\lambda_{S}=0.83$, respectively. In Fig. 8 and 9, there
are four subplots to show the impact of $\gamma$ given $\Gamma$ on the
simulation outcomes. In the following, the meaning of the subplots is
explained.

In each subplot, the discount factor $\gamma$ is varied on the x-axis from $0$
to $0.9$ in steps of $0.1$, while the surplus sharing parameter $\Gamma$ is
varied on the y-axis from $0.1$ to $0.9$ in steps of $0.1$. The top left
subplot displays contours representing the headquarters’ profits resulting
from the simulation. The bottom left subplot depicts the “first-best
performance indicator” for the fuzzy Q-learning agents, which is defined by
$\Pi_{HQ}/\Pi_{HQ}^{*}$, i.e., the headquarters’ profit obtained from the
simulation is normalized by the highest feasible profit that can be achieved.
The first-best performance indicator can serve as a relative indicator for the
profit-effectiveness of the decision-making behavior of the fuzzy Q-learning
agents; the higher the value, the better the profitability for the
headquarters. For example, if the first-best performance indicator reaches
one, then divisions make optimal investment decisions and, in this way, the
divisions as well as the headquarters achieve the maximum profit.

Another performance indicator is visualized in the bottom right subplot. This
performance indicator reflects how much better the fuzzy Q-learning agents
perform than fully rational utility maximizers do. This called “second-best
performance indicator” is formalized by the relative change between $\Pi_{HQ}$
and $\Pi_{HQ}^{sb}$ dividing by $\Pi_{HQ}^{sb}$. Thus, the higher the relative
change, the higher the performance of fuzzy Q-learning agents.

Lastly, and more importantly, the so-called “baseline performance indicator”,
which is displayed in the top right subplot, is based on the relative change
between $\Pi_{HQ}^{\Gamma}$ and $\Pi_{HQ}^{baseline}$ dividing by
$\Pi_{HQ}^{baseline}$, where $\Pi_{HQ}^{\Gamma}$ denotes the headquarters’
profit resulting from scenario where the headquarters uses $\Gamma$, while
$\Pi_{HQ}^{baseline}$ describes the headquarters’ profit resulting from
scenario where the headquarters applies $\Gamma^{sb}$. The scenario with
$\Gamma^{sb}$ is called the “baseline scenario”. So in a baseline scenario,
the headquarters sets the bargaining power $\Gamma$ according to Eq. 21.
Hence, this indicator provides information about how much better fuzzy
Q-learning agents given $\Gamma$ perform than fuzzy Q-learning agents given
$\Gamma^{sb}$.

In order to find out, if there is a significant difference between
$\Pi_{HQ}^{\Gamma}$ and $\Pi_{HQ}^{baseline}$, Welch’s t-tests and Wilcoxon
rank-sum tests are applied.161616 Welch’s (1947) t-test and Wilcoxon rank-sum
test (Mann and Whitney 1947) are widely common statistical hypothesis tests.
In the default case, the first one tests the equality of the means for two
independent normally distributed samples with unequal and unknown variances,
while the second one, a nonparametric test, tests the null hypothesis that the
distributions of two independent samples differ by a location shift of zero
or, roughly spoken, that they have the same distribution. Please be aware that
the p-values of hypothesis tests go quickly to zero when very large samples
($10,000$ observations and more) are evaluated, the null hypotheses become
statistically significant because the standard errors become extremely small
(Lin et al. 2013). To mitigate this phenomenon because, in the simulation
study, $10,000$ simulation runs are carried out and, additionally, in each
simulation run, the last $100$ observations are recorded for evaluating, one-
tailed Welch’s t-tests and Wilcoxon rank-sum tests are used with a positive
hypothesized mean difference (hereafter abbreviated to $d_{H}$). Concretely,
the null hypotheses are given by
$mean(\Pi_{HQ}^{\Gamma})-mean(\Pi_{HQ}^{baseline})\geq d_{H}$. Note that the
Welch’s t-test is designed for normally distributed samples but, due to the
central limit theorem, it can be assumed that this also holds for the
headquarters’ profits which are approximately normally distributed.

In order to find out whether the headquarters’ profits differ significantly in
terms of their means (see Welch’s t-tests) or their distribution (see Wilcoxon
rank-sum tests), the hypothesized mean difference $d_{H}$ is set to
$\Pi_{HQ}^{baseline}/100$, which can be interpreted as $1\%$ of the
headquarters’ profit achieved in the baseline scenario. Hence, whenever the
t-test result is statistically significant, fuzzy Q-learning agents in the
$\Gamma$ scenario perform at least $1\%$ better than fuzzy Q-learning agents
in the baseline scenario given a standard significance level of $0.05$. To
distinguish the test results, the relative changes are colored as follows: If
the p-value of the t-test (rank-sum test) is greater than $0.05$, the cell
turns yellow (blue). If both are statistically significant, the cell is green;
otherwise the cell is white.

Figure 8: Results for symmetric marginal cost parameters ($\lambda_{S}=0.5$
and $\lambda_{B}=0.5$) in the scenario $3$: (a) absolute performance of the
headquarters’ profit resulting from the simulation, (b) relative performance
change and statistical hypothesis testing of $\Pi_{HQ}$ compared to the
simulation output from the baseline scenario with $\Gamma^{sb}=0.5$, (c)
relative performance of $\Pi_{HQ}$ compared to $\Pi_{HQ}^{*}$, and (d)
relative performance change of $\Pi_{HQ}$ compared to $\Pi_{HQ}^{sb}$. Figure
9: Results for non-symmetric marginal cost parameters ($\lambda_{S}=0.83$ and
$\lambda_{B}=0.17$) in the scenario $3$: (a) absolute performance of the
headquarters’ profit resulting from the simulation, (b) relative performance
change and statistical hypothesis testing of $\Pi_{HQ}$ compared to the
simulation output from the baseline scenario with $\Gamma^{sb}=0.1$, (c)
relative performance of $\Pi_{HQ}$ compared to $\Pi_{HQ}^{*}$, and (d)
relative performance change of $\Pi_{HQ}$ compared to $\Pi_{HQ}^{sb}$.

According to Fig. 8, non-myopic fuzzy Q-learning agents reach the highest
profits, but only as long as they have equal bargaining power. As soon as the
distribution of bargaining power changes, the headquarters’ profit decreases,
whereby $\Pi_{HQ}$ declines faster for non-myopic agents because they react
more strongly to a non-symmetrical surplus sharing rule than myopic agents do.
Further, the efficiency of the fuzzy Q-learning agents can be seen in the
bottom contour plots. While the first-best performance indicator points out
that the profit-effectiveness of the headquarters is quite high, especially on
the green contours (cf. the bottom left subplot in Fig. 8), the second-best
performance indicator reflects that, in most parameter constellations, the
decision-making behavior of the fuzzy Q-learning agents is much better than
the predicted behavior of fully rational utility maximizers (cf. the bottom
right subplot in Fig. 8). Finally, tests for statistical significance are
carried out for each parameter constellation (see the top right subplot in
Fig. 8). Based on the test results, it can be concluded that, for symmetric
marginal cost parameters, a deviation from the theory of subgame perfection
equilibrium leads to worse results. In this case, it is more effective to
follow the recommendations based on the transfer pricing literature, e.g.,
Ewert and Wagenhofer (2014), Wagner (2008), Göx and Schiller (2006), and give
both divisions equal bargaining power.

Things look different in scenarios with non-symmetric marginal cost
parameters. According to the simulation outputs for $\lambda_{S}=0.83$ (cf.
top right subplot in Fig. 9), it is obtained that, for myopic agents with a
foresight between $0$ and $0.4$, the surplus sharing parameter should also be
set to $\Gamma^{sb}$, but, for non-myopic fuzzy Q-learning agents with
$\gamma\geq 0.5$, a deviation from the recommendations leads to better
results. Overall, there are $16$ ($14$) scenarios in which the Welch’s t-test
(Wilcoxon rank-sum test) indicates a significant difference between
$\Pi_{HQ}^{\Gamma}$ and $\Pi_{HQ}^{baseline}$. In order to get a reference
value for the distribution ratio of the bargaining power for non-myopic fuzzy
Q-learning agents, the weighted arithmetic mean of $\Gamma$ is simply
calculated, i.e., $\sum_{\Gamma=0.1}^{0.9}\sum_{\gamma=0}^{0.9}\Gamma\cdot
BPI[\Gamma,\gamma]/\sum_{\Gamma=0.1}^{0.9}\sum_{\gamma=0}^{0.9}BPI[\Gamma,\gamma]$,
where the baseline performance indicator $BPI$ is significant. The weighted
arithmetic mean of $\Gamma$ is $0.33$, which can be considered as a good
choice for fuzzy Q-learning agents with a foresight between $0.5$ and $0.9$.
Accordingly, the supplying division receives one third of the bargaining power
from the headquarters, while the buying division gets two thirds. This rule of
thumb can be particularly relevant because, in general, the headquarters
cannot optimally determine the bargaining power, as, e.g., information
required to solve the three-stage decision problem is often missing.

###  5.4 Sensitivity analysis on the variability of state variables and
implementation of exploration policies

Finally, an extensive sensitivity analysis is performed to check whether the
simulation results are robust in terms of the variability of the state
variables and the implementation of other exploration policies. In order to
evaluate the robustness of the analysis, the environment parameters are
adapted so that stochastic fluctuations can now occur. For this purpose, the
standard deviations of the state variables $\sigma[\theta_{S}]$ and
$\sigma[\theta_{B}]$ are set from $0$ to $5$ and $10$, respectively. In
addition to the Boltzmann exploration policy, the $\epsilon$-greedy and the
upper confidence bound exploration policy, which are introduced in Sec. 4, are
applied. Recall that the $\epsilon$-greedy exploration policy is given in a
natural manner and, therefore, can be seen as a benchmark for other more
sophisticated exploration policies (Tijsma et al. (2016) compare
$\epsilon$-greedy, Boltzmann, and UCB policy and evaluate their performance
for Q-learning agents).

Fig. 10-19 in the appendix show the results from the sensitivity analysis. The
contour plots depict the absolute performance of the headquarters’ profit
resulting from the simulation. Note, while the means of $\Pi_{HQ}$ vary from
scenario to scenario, the scale of measure is not set uniformly for all
scenarios. This has the effect that the decision-making behavior of the fuzzy
Q-learning agents can be better derived, since the headquarters’ profit
depends positively on the volatility of the markets (see Eq. 13 and 25). For
example, if $\sigma[\theta_{S}]=\sigma[\theta_{B}]=5$, then the additional
profit is $(5^{2}+5^{2})/(2\cdot 12)=2.08$ and, for
$\sigma[\theta_{S}]=\sigma[\theta_{B}]=10$, the headquarters benefits from the
volatility of the markets by a value of $8.33$.

In the case of symmetric marginal cost parameters (cf. Fig. 10), all three
exploration policies show a similar picture, whereby fuzzy Q-learning using
the $\epsilon$-greedy exploration policy performing worst and fuzzy Q-learning
using the Boltzmann exploration policy performing best. When comparing the
columns in Fig. 10, fuzzy Q-learning agents exhibit almost a similar decision-
making behavior, i.e., they are insensitive to environmental fluctuations.
According to Fig. 15, there is no parameter constellation that leads to a
better performance than in the baseline scenario with $\Gamma=0.5$.

In the case of non-symmetric marginal cost parameters (cf. Fig. 11-19), the
learning behavior of the fuzzy Q-learning agents also seems to be robust
against the volatilities of the markets. More importantly, the simulation
outputs for $\lambda_{S}>0.5$ indicate that following the recommendations for
setting the surplus sharing parameter $\Gamma$ according to Eq. 21 does not
always lead to the best result. Especially for non-myopic fuzzy Q-learning
agents using $\epsilon$-greedy, UCB or Boltzmann exploration policy, the
headquarters obtains higher profits when the seller is assigned a greater
bargaining power. Thus, the higher the marginal cost parameter $\lambda_{S}$,
the better it is for all parties involved that the seller gets a higher
bargaining power. In such a case, the headquarters should assign the supplying
division a bargaining power based on the calculation of the weighted
arithmetic mean. It is also interesting to observe that myopic fuzzy
Q-learning agents using a UCB policy contribute to higher profits, if the
headquarters assigns the supplying division a lower bargaining power.

##  6 Summary and conclusive remarks

One main reason for studies in the field of specific investments under
transfer pricing is to find conditions which provide first-best solutions for
the hold-up problem. However, the transfer pricing problem has been developed
in the context where divisions are fully individual rational utility
maximizers. The underlying assumptions are rather heroic and, in particular,
how divisions process information under uncertainty, do not perfectly match
with human decision-making behavior. Therefore, this paper relaxes key
assumptions and studies whether cognitively bounded agents achieve the same
results as fully rational utility maximizers and, in particular, whether the
recommendations on managerial-compensation arrangements and bargaining
infrastructures are designed to maximize headquarters’ profit in such a
setting. For this purpose, an agent-based variant of negotiated transfer
pricing with cognitively bounded agents is analyzed.

This paper limits the research to five parameters: (1) the marginal cost
parameter in the divisional investment cost function, (2) the surplus sharing
parameter in the divisions’ incentive compatibility, (3) the discount factor
in the fuzzy Q-learning method, (4) the standard deviation of the state
variables in regard to the volatility of the markets, and (5) the exploration
policy for the action-selection of investments and, in order to produce new
theoretical insights, these parameters are systematically vary in a controlled
setting.

Based on an agent-based simulation with fuzzy Q-learning agents, it is shown
that (1) in case of symmetric marginal cost parameters, myopic fuzzy
Q-learning agents invest only as much as in the classic hold-up problem
(second-best solution), while non-myopic fuzzy Q-learning agents invest
optimally (first-best solution). (2) in scenarios with non-symmetric marginal
cost parameters, $\lambda_{S}>\lambda_{B}$, the supplying division always
invests in the value range of the second-best solution, regardless of its
foresight, while the performance of the buying division depends crucially on
the level of foresight, but first-best solutions are not reached in this
setting. (3) the simulation output shows that varying the surplus sharing
parameter does not lead to significantly better performance in scenarios with
symmetric marginal cost parameters. However, in scenarios with non-symmetric
marginal cost parameters, the choice of the surplus sharing parameter can lead
to a significant improvement in the divisions’ performance than in the case of
the recommended surplus sharing parameters. Therefore, a deviation from the
recommended surplus sharing parameter can increase the headquarters’ profit.
The higher the degree of divisions’ foresight, the higher the performance
output, if the headquarters assigns the seller a higher bargaining power than
the theory of subgame perfection equilibrium recommends. (4) according to the
sensitivity analysis, the simulation results are robust with respect to the
variability of the state variables and the implementation of $\epsilon$-greedy
and upper confidence bound exploration policy.

The simulation outcomes show that learning by fuzzy Q-learning can be
beneficial for economic agents, but whether fuzzy Q-learning matches perfectly
with human decision-making behavior is, of course, an empirical question.
Beyond that, the findings are based on an agent-based simulation and,
therefore, the individual learning behavior of the fuzzy Q-learning agents
depends in particular on how the discount factor and the exploration policy
are determined. Nonetheless, learning methods from the field of reinforcement
learning, such as fuzzy Q-learning, could provide a further clue as to why
decentralized firms have the intuition to assign each division equal
bargaining power, even if they have different marginal costs parameters.

Finally, future research could examine the following points: (1) the research
focuses on a decentralized firm with only two competing divisions. However, it
is also interesting to investigate other decentralized firm settings with more
than two divisions, since analytical results are generally not available for
such settings. (2) the use of more complicated membership functions, such as
those implemented with Gaussian or S-shapes. Such are very popular in the
fuzzy logic literature and have properties, such as being differentiable
everywhere, but they are more computationally expensive than triangular
membership functions. (3) future research could also examine a different
sequence of events. It would be interesting to know to what extent the
simulation results change, if, for example, the state variables are not
observed until after the negotiation, or that the investments are also subject
to stochastic fluctuations.

## Appendix: Sensitivity analysis conducted according to scenario 4

Figure 10: Absolute performance of the headquarters’ profit of different
exploration policies and standard deviations of state variables given
$\lambda_{S}=\lambda_{B}=0.5$. Figure 11: Absolute performance of the
headquarters’ profit of different exploration policies and standard deviations
of state variables given $\lambda_{S}=0.58$ and $\lambda_{B}=0.42$.

Figure 12: Absolute performance of the headquarters’ profit of different
exploration policies and standard deviations of state variables given
$\lambda_{S}=0.66$ and $\lambda_{B}=0.34$. Figure 13: Absolute performance of
the headquarters’ profit of different exploration policies and standard
deviations of state variables given $\lambda_{S}=0.75$ and $\lambda_{B}=0.25$.

Figure 14: Absolute performance of the headquarters’ profit of different
exploration policies and standard deviations of state variables given
$\lambda_{S}=0.83$ and $\lambda_{B}=0.17$. Figure 15: Results of the baseline
performance indicator for $\Gamma^{sb}=0.5$ of different exploration policies
and standard deviations of state variables given
$\lambda_{S}=\lambda_{B}=0.5$.

Figure 16: Results of the baseline performance indicator for $\Gamma^{sb}=0.4$
of different exploration policies and standard deviations of state variables
given $\lambda_{S}=0.58$ and $\lambda_{B}=0.42$. Figure 17: Results of the
baseline performance indicator for $\Gamma^{sb}=0.3$ of different exploration
policies and standard deviations of state variables given $\lambda_{S}=0.66$
and $\lambda_{B}=0.34$.

Figure 18: Results of the baseline performance indicator for $\Gamma^{sb}=0.2$
of different exploration policies and standard deviations of state variables
given $\lambda_{S}=0.75$ and $\lambda_{B}=0.25$. Figure 19: Results of the
baseline performance indicator for $\Gamma^{sb}=0.1$ of different exploration
policies and standard deviations of state variables given $\lambda_{S}=0.83$
and $\lambda_{B}=0.17$.

## References

  * Agogino and Tumer (2005) Agogino AK, Tumer K (2005) Quicker q-learning in multi-agent systems. URL https://ntrs.nasa.gov/api/citations/20050182925/downloads/20050182925 .pdf
  * Anctil and Dutta (1999) Anctil RM, Dutta S (1999) Negotiated transfer pricing and divisional vs. firm-wide performance evaluation. The Accounting Review 74(1):87–104
  * Arthur (1991) Arthur WB (1991) Designing economic agents that act like human agents: A behavioral approach to bounded rationality. The American Economic Review 81(2):353–359
  * Auer et al. (2002) Auer P, Cesa-Bianchi N, Fischer P (2002) Finite-time analysis of the multiarmed bandit problem. Machine Learning 47(2-3):235–256
  * Aumann (1989) Aumann R (1989) Game theory. In: Eatwell J, Milgate M, Newman P (eds) Game Theory, New York: Norton, pp 1–53
  * Baldenius (2000) Baldenius T (2000) Intrafirm trade, bargaining power, and specific investments. Review of Accounting Studies 5(1):27–56
  * Baldenius et al. (1999) Baldenius T, Reichelstein S, Sahay SA (1999) Negotiated versus cost-based transfer pricing. Review of Accounting Studies 4(2):67–91
  * Cesa-Bianchi et al. (2017) Cesa-Bianchi N, Gentile C, Lugosi G, Neu G (2017) Boltzmann exploration done right. In: Advances in Neural Information Processing Systems, pp 6284–6293
  * Collins (2019) Collins AGE (2019) Reinforcement learning: bringing together computation and cognition. Current Opinion in Behavioral Sciences 29:63–68
  * Cook (1955) Cook PW (1955) Decentralization and the transfer-price problem. The Journal of Business 28(2):87–94
  * Davis et al. (2007) Davis JP, Eisenhardt KM, Bingham CB (2007) Developing theory through simulation methods. Academy of Management Review 32(2):480–499
  * Dearden et al. (1998) Dearden R, Friedman N, Russell S (1998) Bayesian q-learning. In: Proceedings of the Fifteenth National Conference on Artificial Intelligence, pp 761–768
  * Eccles (1985) Eccles RG (1985) The transfer pricing problem: A theory for practice. Lexiongton, MA: Lexington Books
  * Eccles and White (1988) Eccles RG, White HC (1988) Price and authority in inter-profit center transactions. American Journal of Sociology 94:17–51
  * Edlin and Reichelstein (1995) Edlin AS, Reichelstein S (1995) Specific investment under negotiated transfer pricing: An efficiency result. Accounting Review 70(2):275–291
  * Epstein (2006) Epstein JM (2006) Agent-based computational models and generative social science. In: Generative social science: Studies in agent-based computational modeling, Princeton University Press, pp 4–46
  * Ewert and Wagenhofer (2014) Ewert R, Wagenhofer A (2014) Interne Unternehmensrechnung. Springer
  * Feltovich (2000) Feltovich N (2000) Reinforcement-based vs. belief-based learning models in experimental asymmetric-information games. Econometrica 68(3):605–641
  * Fudenberg and Tirole (1991) Fudenberg D, Tirole J (1991) Perfect bayesian equilibrium and sequential equilibrium. Journal of Economic Theory 53(2):236–260
  * Georgila et al. (2014) Georgila K, Nelson C, Traum D (2014) Single-agent vs. multi-agent techniques for concurrent reinforcement learning of negotiation dialogue policies. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, vol 1, pp 500–510
  * Glorennec (1994) Glorennec PY (1994) Fuzzy q-learning and dynamical fuzzy q-learning. In: Proceedings of 1994 IEEE 3rd International Fuzzy Systems Conference, IEEE, pp 474–479
  * Göx and Schiller (2006) Göx RF, Schiller U (2006) An economic perspective on transfer pricing. Handbooks of Management Accounting Research 2:673–695
  * Guerrero and Axtell (2011) Guerrero OA, Axtell RL (2011) Using agentization for exploring firm and labor dynamics. In: Osinga S, Hofstede GJ, Verwaart T (eds) Emergent results of artificial economics, Springer Heidelberg, pp 139–150
  * Hirshleifer (1956) Hirshleifer J (1956) On the economics of transfer pricing. The Journal of Business 29(3):172–184
  * Kofinas et al. (2018) Kofinas P, Dounis A, Vouros G (2018) Fuzzy q-learning for multi-agent decentralized energy management in microgrids. Applied Energy 219:53–67
  * Küpper et al. (2013) Küpper HU, Friedl G, Hofmann C, Hofmann Y, Pedell B (2013) Controlling: Konzeption, Aufgaben, Instrumente, vol 6. Schäffer-Poeschel Verlag Stuttgart
  * Laffont and Martimort (2009) Laffont JJ, Martimort D (2009) The theory of incentives: the principal-agent model. Princeton University Press
  * Leitner and Behrens (2015) Leitner S, Behrens D (2015) On the efficiency of hurdle rate-based coordination mechanisms. Mathematical and Computer Modelling of Dynamical Systems 21(5):413–431
  * Lin et al. (2013) Lin M, Lucas Jr HC, Shmueli G (2013) Research commentary: too big to fail: large samples and the p-value problem. Information Systems Research 24(4):906–917
  * Mann and Whitney (1947) Mann HB, Whitney DR (1947) On a test of whether one of two random variables is stochastically larger than the other. The Annals of Mathematical Statistics 18(1):50–60
  * Milgrom and Roberts (1992) Milgrom PR, Roberts JD (1992) Economics, organization and management. Prentice-Hall
  * Niv (2009) Niv Y (2009) Reinforcement learning in the brain. Journal of Mathematical Psychology 53(3):139–154
  * Osborne and Rubinstein (1994) Osborne MJ, Rubinstein A (1994) A course in game theory. MIT Press
  * Pfaff and Pfeiffer (2004) Pfaff D, Pfeiffer T (2004) Verrechnungspreise und ihre formaltheoretische Analyse: Zum State of the Art. Die Betriebswirtschaft 64(3):296
  * Pfeiffer (2004) Pfeiffer T (2004) The value of information in the hold-up problem. German Economic Review 5(2):177–203
  * Pfeiffer and Wagner (2007) Pfeiffer T, Wagner J (2007) Die Rekonstruktion interner Märkte, das Dilemma der pretialen Lenkung und spezifische Investitionsprobleme. Schmalenbachs Zeitschrift für Betriebswirtschaftliche Forschung 59(8):958–981
  * Powell and Ryzhov (2012) Powell WB, Ryzhov IO (2012) Optimal learning, vol 841. John Wiley & Sons
  * Rahimiyan and Mashhadi (2006) Rahimiyan M, Mashhadi HR (2006) Modeling the supplier agent’s risk strategy based on fuzzy logic combined with the q-learning algorithm. In: 2006 International Conference on Computational Intelligence and Security, IEEE, vol 1, pp 159–163
  * Richiardi et al. (2006) Richiardi MG, Leombruni R, Saam NJ, Sonnessa M (2006) A common protocol for agent-based social simulation. Journal of Artificial Societies and Social Simulation 9(1)
  * Ronen and Balachandran (1988) Ronen J, Balachandran KR (1988) An approach to transfer pricing under uncertainty. Journal of Accounting Research 26(2):300–314
  * Rubinstein (1982) Rubinstein A (1982) Perfect equilibrium in a bargaining model. Econometrica: Journal of the Econometric Society 50(1):97–109
  * Sandholm and Crites (1996) Sandholm TW, Crites RH (1996) Multiagent reinforcement learning in the iterated prisoner’s dilemma. Biosystems 37(1-2):147–166
  * Schmalenbach (1908/1909) Schmalenbach E (1908/1909) Über Verrechnungspreise. Zeitschrift für Handelswissenschaftliche Forschung 3(9):165–185
  * Sebenius (1992) Sebenius JK (1992) Negotiation analysis: A characterization and review. Management Science 38(1):18–38
  * Simon (1955) Simon HA (1955) A behavioral model of rational choice. The Quarterly Journal of Economics 69(1):99–118
  * Simon (1979) Simon HA (1979) Rational decision making in business organizations. The American Economic Review 69(4):493–513
  * Sutton and Barto (2018) Sutton RS, Barto AG (2018) Reinforcement learning: An introduction. MIT Press
  * Tesauro and Kephart (2002) Tesauro G, Kephart JO (2002) Pricing in agent economies using multi-agent q-learning. Autonomous Agents and Multi-Agent Systems 5(3):289–304
  * Thrun (1992) Thrun SB (1992) Efficient exploration in reinforcement learning. Technical Report CS-CMU-92-102, School of Computer Science
  * Tijsma et al. (2016) Tijsma AD, Drugan MM, Wiering MA (2016) Comparing exploration strategies for q-learning in random stochastic mazes. In: Symposium Series on Computational Intelligence, IEEE, pp 1–8
  * Vaysman (1998) Vaysman I (1998) A model of negotiated transfer pricing. Journal of Accounting and Economics 25(3):349–384
  * Wagenhofer (1994) Wagenhofer A (1994) Transfer pricing under asymmetric information: An evaluation of alternative methods. European Accounting Review 3(1):71–103
  * Wagner (2008) Wagner J (2008) Corporate Governance Strukturen zur Lösung von Koordinations- und Anreizproblemen bei Auftreten spezifischer Investitionsprobleme. PhD thesis, Universität Wien
  * Wall (2016) Wall F (2016) Agent-based modeling in managerial science: an illustrative survey and study. Review of Managerial Science 10(1):135–193
  * Waltman and Kaymak (2008) Waltman L, Kaymak U (2008) Q-learning agents in a cournot oligopoly model. Journal of Economic Dynamics and Control 32(10):3275–3293
  * Watkins and Dayan (1992) Watkins CJ, Dayan P (1992) Q-learning. Machine Learning 8(3-4):279–292
  * Watkins (1989) Watkins CJCH (1989) Learning from delayed rewards. PhD thesis, King’s College, Cambridge
  * Welch (1947) Welch BL (1947) The generalization of ’student’s’ problem when several different population variances are involved. Biometrika 34(1/2):28–35
  * Williamson (1979) Williamson OE (1979) Transaction-cost economics: the governance of contractual relations. The Journal of Law and Economics 22(2):233–261
  * Williamson (1985) Williamson OE (1985) The economic institutions of capitalism. New York: Free Press
  * Young (2001) Young HP (2001) Individual strategy and social structure: An evolutionary theory of institutions. Princeton University Press
  * Zimmermann (2011) Zimmermann HJ (2011) Fuzzy set theory and its applications. Springer Science & Business Media