# Multi-unit Double Auctions: Equilibrium Analysis and Bidding Strategy using
DDPG in Smart-grids

Sanjay Chandlekar
Machine Learning Lab, IIIT Hyderabad
Hyderabad, India
<EMAIL_ADDRESS>
&Easwar Subramanian
TCS Innovation Labs
Hyderabad, India
<EMAIL_ADDRESS>
Sanjay Bhat
TCS Innovation Labs
Hyderabad, India
<EMAIL_ADDRESS>
&Praveen Paruchuri
Machine Learning Lab, IIIT Hyderabad
Hyderabad, India
<EMAIL_ADDRESS>
&Sujit Gujar
Machine Learning Lab, IIIT Hyderabad
Hyderabad, India
<EMAIL_ADDRESS>

###### Abstract

Periodic double auctions (PDA) have applications in many areas such as in
e-commerce, intra-day equity markets, and day-ahead energy markets in _smart-
grids_. While the trades accomplished using PDAs are worth trillions of
dollars, finding a reliable bidding strategy in such auctions is still a
challenge as it requires the consideration of future auctions. A participating
buyer in a PDA has to design its bidding strategy by planning for current and
future auctions. Many equilibrium-based bidding strategies proposed are
complex to use in real-time. In the current exposition, we propose a scale-
based bidding strategy for buyers participating in PDA. We first present an
equilibrium analysis for single-buyer single-seller multi-unit single-shot
k-Double auctions. Specifically, we analyze the situation when a seller and a
buyer trade two identical units of quantity in a double auction where both the
buyer and the seller deploy a simple, scale-based bidding strategy. The
equilibrium analysis becomes intractable as the number of participants
increases. To be useful in more complex settings such as wholesale markets in
smart-grids, we model equilibrium bidding strategy as a learning problem. We
develop a _deep deterministic policy gradient (DDPG)_ based learning strategy,
DDPGBBS, for a participating agent in PDAs to suggest an action at any auction
instance. DDPGBBS, which empirically follows the obtained theoretical
equilibrium, is easily extendable when the number of buyers/sellers increases.
We take Power Trading Agent Competition’s (PowerTAC) wholesale market PDA as a
testbed to evaluate our novel bidding strategy. We benchmark our DDPG based
strategy against several baselines and state-of-the-art bidding strategies of
the PowerTAC wholesale market PDA and demonstrate the efficacy of DDPGBBS
against several benchmarked strategies.

_Keywords_ Multi-unit Periodic Double Auction $\cdot$ Equilibrium Analysis
$\cdot$ Bidding Strategy $\cdot$ DDPG for Learning Equilibrium Strategy

##  1 Introduction

An auction is a process of buying or selling goods or items Auction-Wikipedia
(2021), and it could be of varied types Y Narahari (2014). For example, a
_forward auction_ consists of one seller and multiple buyers who submit their
_bids_ ; a _reverse auction_ has one buyer and multiple sellers submitting
their _asks_ ; a _double auction_ comprising of multiple buyers and sellers
placing their bids and asks. Double auctions are extensively used in the real
world to trade stocks, energy, spectrum, and many other goods and services.
Stock Exchanges are one example where a double auction generally takes place
and involves trillions of dollars daily trades. The other prominent domain
where the double auction plays a significant role is _smart-grids_ Safak
(2014).

Multiple power generating companies (GenCos) and energy brokers trade energy
in the wholesale market through double auction in smart grids. Nord Pool,
which runs the leading power market in Europe, showed trades of $995$ $TWh$ of
volume, with close to 60% of the volume traded using APIs Nord Pool (2020).
Clearly, a bidding strategy that can optimize the cost of energy brokers even
by a small amount would significantly improve their profits and bring in more
efficiency to the system. While such double auctions are common in the real
world, designing an optimal bidding strategy that an energy broker (buyer) can
use in real-time applications is still challenging. The problem gets further
complicated in settings where periodic double auctions are deployed. In PDAs,
the buyer needs to participate in a series of auctions, and therefore, a
bidding strategy involves planning across current and future auctions. For
example, an energy broker participating in day-ahead markets can procure
energy for a delivery time-slot at any prior time within 24 hours.

Motivated with such smart-grid energy trading, we focus on scaling-based
bidding strategies for the following reasons: - (i) they are the most natural
when the brokers need to submit bids in real-time. (ii) as compared to solving
differential equations Vetsikas (2008) or fictitious play formulations Shi
(2010) to devise strategies, which are complex in nature, scale-based
strategies are easy to implement. Hence, the primary reason to use scale-based
strategies is their simplicity. In settings like smart-grids, simple
strategies are desired when an agent needs to bid in real-time. Ghosh et al.
Susobhan (2020) already designed a scale-based bidding strategy for single
unit double auction and claim to have a good performance, making them runner-
up in the PowerTAC 2018 tournament. We study a two-unit double auction with a
single seller and a single buyer deploying scaling-based strategies. We
characterize Nash equilibria for scaling-based strategies for the buyer and
the seller when they trade two units ($m=2$) of identical and indivisible
items at each instance of a $k$-double auction. Characterizing equilibrium
becomes intractable beyond $m=2$. Hence, we address the issue of whether an
intelligent agent (buyer) can learn the equilibrium strategy.

To address the issue of learning equilibrium strategy, we use techniques from
_reinforcement learning_ and, more specifically, we develop a deep
deterministic policy gradient-based bidding strategy. The strategy thus
developed is then tested on a smart-grid wholesale market ecosystem called
PowerTAC, which is an efficient implementation of the real-world smart-grid
Ketter (2020). PowerTAC replicates the crucial elements of smart-grids; more
importantly, it has PDA for energy trading in the wholesale energy market.
Past works that were used in the PowerTAC framework include MDP-based bidding
strategies used in TacTex Urieli (2014) and VidyutVanika (called MDPLCPBS)
Susobhan (2020), an MCTS based formulation in SPOT Chowdhury (2018) among
others. These prior strategies discretize the action space, which is typically
continuous in nature. In addition, except SPOT, most other works treat the
bidding process as a single unit auction. Whereas, in this work, we consider
the setting of a two-unit auction (dividing the required quantity into two
equal parts) as we believe it helps in bringing down the clearing price of an
auction With this background, we design a Deep-Deterministic-Policy-Gradient
(DDPG) based bidding strategy – DDPGBBS for double auctions. We chose DDPG, as
it has proved to be very successful in a variety of _planning_ tasks Lillicrap
(2016). DDPGBBS learns to bid optimally with the help of feedback from the
auction environment.

To test the efficacy of DDPGBBS to learn equilibrium strategies, first, we
conduct validation experiments to confirm if it learns the known equilibria.
We observe that it can approximate the NE under the controlled experimental
setting for all the known cases. We designed a simulator comprising one buyer,
one seller, and a two-unit double auction to perform validation experiments.
We model the seller to follow its NE strategy and train the agent (buyer)
against it.

Post verifying the ability to learn equilibrium bidding strategy with DDPGBBS;
we extend it to model general PDA where there is no restriction on the number
of participants or the number of units traded in the auction. As there are
multiple opportunities for the buyer to buy energy for a particular delivery
time-slot, we introduce _proximity_ to the delivery time-slot into our state-
space formulation. For training and validating DDPGBBS for such real-world
scenarios, we leverage the PowerTAC PDA simulation environment to conduct all
our experiments instead of building our simulator. Many researchers have
attempted to design a bidding strategy for the PowerTAC PDA; previous bidding
strategies like MCTS based strategy of SPOT Chowdhury (2018), MDPLCPBS of
VidyutVanika Susobhan (2020) and MDP based strategy of TacTex Urieli (2014)
are proven to be effective in the annual PowerTAC tournament. However, they do
not perform game-theoretical equilibrium analysis for their strategies except
VidyutVanika. Although VidyutVanika demonstrated that their MDPLCPBS follows
theoretical Nash equilibrium, their work was limited to single-unit demand.

To the best of our knowledge, we are the first to utilize the policy gradient-
based RL algorithm, enabling us to work with PowerTAC’s continuous state and
action space more effectively. In summary, our contributions are as follows:

  1. 1.

We analyze Bayesian Nash equilibrium strategies for one buyer and one seller
in multi-unit auctions, where both buyer and seller trade for two identical
items practicing scale-based bidding strategies. We present an analysis for
each of the four possible scenarios depending on whether the buyer and
seller’s scale factors are the same for both units or different.

  2. 2.

We formalize a DDPG based bidding strategy (DDPGBBS) for multi-unit double
auction and validate that it learns the theoretical equilibrium.

  3. 3.

We extend DDPGBBS to work effectively in PDAs such as the PowerTAC wholesale
market. We then demonstrate its efficacy against baseline and state-of-the-art
bidding strategies for PowerTAC.

##  2 Related Work

Chatterjee and Samuelson Chatterjee (1983) proposed a Nash Equilibrium
analysis for a single buyer and single seller with the general distribution of
valuations and constructed NE strategy for a special case of uniform
valuations. Satterthwaite and Williams Satterthwaite (1989) address the
problem of finding the existence of a multiplicity of equilibria for the
$k$-double auction for a generic class of traders’ valuations; they too
propose the equilibrium strategies in the form of differential equations and
then examine the efficiency of the proven equilibrium. Vetsikas Vetsikas
(2014) attempts to find equilibrium strategies for multi-unit sealed bid
auction for $m^{th}$ and $(m+1)^{th}$ price sealed bid auction.

As for bidding strategies in PowerTAC PDA, Kaute et al. Rodrigue (2013) uses
an MDP-based bidding strategy in their broker AstonTAC, which outputs the
ratio of remaining energy that needs to be ordered, while limit-price comes
from the Non-Homogeneous Hidden Markov Models (NHHMM). Urieli and Stone Urieli
(2014) propose an MDP-based strategy inspired by Tesauro and Bredin’s bidding
strategy Tesauro (2002) for broker TacTex, and they solve this MDP using
dynamic programming. Ghosh et al. Susobhan (2020) improve upon TacTex’s
bidding strategy to use in the VidyutVanika broker; additionally, they provide
theoretical analysis for their MDP-based strategy. Chowdhury et al. Chowdhury
(2018) Chowdhury (2018) use a Monte Carlo Tree Search (MCTS) based strategy on
top of REPTree based price predictor coupled with heuristics to determine the
optimal bid prices. Ozdemir and Unland Özdemir (2015) use an adaptive
Q-learning-based strategy in their broker AgentUDE. TUC_TAC Orfanoudakis
(2021) too uses an MCTS based bidding strategy inspired by SPOT in the
wholesale market.

The existing theoretical equilibrium for multi-unit double auction are complex
and may not be readily helpful in real-world double auctions. In contrast, our
scale-based bidding strategy is easy to implement for such auctions.
Furthermore, none of the previous works attempt to design a bidding strategy
for multi-unit double auctions backed up by game-theoretic analysis; we show
that our novel DDPGBBS based algorithm works effectively in double auctions
while closely following the theoretical equilibria.

##  3 Preliminaries

We define here all the terminologies and notations used in this paper.

###  3.1 k-Double Auction

k-double auction is a type of auction for buying and selling resources, where
potential buyers submit their bids and potential sellers submit their asks to
the auctioneer. The auctioneer receives all the bids and asks and determines
each player’s _clearing quantity_ and the _clearing price_ using a specific
allocation rule and payment rule.

  * Clearing Price

A clearing price is a price at which the auctioneer clears the market after
matching potential bids of the buyers with the potential asks of the seller.

  * Clearing Quantity

A clearing quantity of a player is the number of items a buyer[seller]
receives[sells] from the auction after clearing.

  * Allocation Rule

An allocation rule determines the quantity bought by each buyer and the
quantity sold by each seller in an auction. Basically, the allocation rule
determines the clearing quantity of each player.

  * Payment Rule

A payment rule determines the payment each player has to make (buyer pays and
seller earns) at the time of auction clearing.

  * $k$-Double Auction

If a buyer and seller participate in a double auction, and if the buyer’s bid
$b$ is higher than the seller’s bid $s$, then the clearing price is given by
$kb+(1-k)s$ for some fixed $k\in[0,1]$.

In this paper, we consider a specific case of _k-double auction_ for _$k$
=0.5_, which we refer to as the average clearing price rule (ACPR). This
implies that the auction clearing price for $k=0.5$ would be an average of the
last clearing bid and the last clearing ask. Additionally, we assume the
uniform pricing rule, which says that all the buyers[sellers] pay[receive] the
same price for a unit quantity.

###  3.2 Notation

This section formally defines the auction setting to be analyzed and the
notations used throughout this paper. In particular, we compute and analyze
the Bayesian Nash equilibrium for a _multi-unit k-double auction (for
$k=0.5$)_ involving one buyer and one seller. The buyer is denoted by $B$, and
the seller by $S$. The Nash equilibrium analysis is carried out for $m=2$
units of identical and indivisible items/quantities.

Let us assume that the true types (true valuations) of buyer $B$ and seller
$S$ are $\theta_{B}$ and $\theta_{S}$, respectively. Both $B$ and $S$ place
two bids/asks in the auction by following _scale-based bidding strategies_
$b_{B}$ and $b_{S}$, respectively. The buyer $B^{\prime}s$ scale based
strategy $b_{B}$ is defined as a strategy in which it places two bids
$b_{B}^{1}=\alpha_{B1}\theta_{B}$ and $b_{B}^{2}=\alpha_{B2}\theta_{B}$; while
seller $S$ places two asks $b_{S}^{1}=\alpha_{S1}\theta_{S}$ and
$b_{S}^{2}=\alpha_{S2}\theta_{S}$ by following strategy $b_{S}$. Here,
$\alpha_{B1}$ and $\alpha_{B2}$ are the scale factors by which buyer $B$
scales its true type (assuming $\alpha_{B1}\geq\alpha_{B2}$), while
$\alpha_{S1}$ and $\alpha_{S2}$ are the scale factors by which seller $S$
scales its true type (assuming $\alpha_{S1}\leq\alpha_{S2}$). Additionally, we
assume $\theta_{B}\sim U[l_{B},h_{B}]$ and $\theta_{S}\sim U[l_{S},h_{S}]$ and
this is a common knowledge; which implies that the true types of the buyer and
seller are sampled from a uniform distribution from the mentioned intervals.
The expected utilities of B and S are denoted by $U_{B}$ and $U_{S}$,
respectively.

###  3.3 Bayesian Nash Equilibrium

In this work, we are characterising an equilibrium in the _game of incomplete
information_ as a player does not know the actual type of the opponent.
Particularly, we are interested in Bayesian Nash equilibrium analysis.

  * Bayesian Nash equilibrium

A Bayesian Nash equilibrium in a Bayesian game $\Gamma$ is defined as, a
profile of strategies $(s_{1}^{*},s_{2}^{*},...,s_{n}^{*})$ is a Bayesian Nash
Equilibrium, if $\forall i\in N$; $\forall s_{i}:\phi_{i}\rightarrow S_{i}$;
$\forall\theta_{i}\in\phi_{i}$,

$u_{i}((s_{i}^{*},s_{-i}^{*})\mid\theta_{i})\geq
u_{i}((s_{i},s_{-i}^{*})\mid\theta_{i})$

That is, $\forall i\in N$; $\forall a_{i}\in S_{i}$;
$\forall\theta_{i}\in\phi_{i}$,

$\mathbb{E_{\theta_{-i}}}[u_{i}(\theta_{i},\theta_{-i},s_{i}^{*}(\theta_{i}),s_{-i}^{*}(\theta_{-i}))]\geq\mathbb{E_{\theta_{-i}}}[u_{i}(\theta_{i},\theta_{-i},a_{i},s_{-i}^{*}(\theta_{-i}))]$

where, $N$ is a set of players, $\theta_{i}$ and $u_{i}(.)$ are true type and
utility of player $i$, respectively.

###  3.4 Markov Decision Process (MDP)

We introduce a DDPG based bidding strategy for double auctions, which is an RL
algorithm and follows the Markov Decision Process (MDP) framework,

  * Markov Decision Process (MDP)

A Markov Decision Process (MDP) is a Reinforcement Learning framework; which
is a tuple represented by $M=(S,A,P,r,\gamma)$, where $S$ denotes the set of
states, $A$ denotes the set of actions, $P$ is the transition probability
function, where $P(s^{\prime}\mid s,a)=P(s_{t+1}=s^{\prime}\mid
s_{t}=s,a_{t}=a)$ is the probability that taking action $a$ in state $s$ at
time $t$ will lead to state $s^{\prime}$ at time t+1, $r$ denotes the reward
function, where $r(s,a)$ is the reward obtained by taking action $a$ in state
$s$, and $\gamma\in[0,1]$ denotes the discount factor.

###  3.5 PowerTAC

In our work, we focus on designing an optimal bidding strategy for PDA.
PowerTAC Ketter (2020) simulates the real-world smart-grid, which also
includes wholesale energy market PDA. PowerTAC simulates a smart-grid for
approximately 60 simulation days, wherein it replicates all major entities of
smart-grids, i.e., various types of customers, power generation companies
(GenCo), Transmission System Operator (TSO), and energy brokers. In any game
in PowerTAC, multiple energy brokers compete against each other across three
markets, namely wholesale, retail, and balancing market. A broker has to buy
or sell energy in the wholesale market to keep supply and demand in balance.
Here, we only focus on the wholesale and balancing markets to benchmark our
DDPGBBS against state-of-the-art and baseline PowerTAC bidding strategies.

PowerTAC wholesale market PDA employs a day-ahead auction, wherein a broker
can purchase energy 24 hours ahead of the delivery time. A total of 24
auctions take place periodically at one-hour intervals for any delivery
timeslot. PowerTAC PDA exercises ACPR with uniform pricing as the payment
rule. In wholesale market PDA, a broker has to compete with other brokers and
the default players in the simulator. During a game, a broker does not know
about other brokers’ types (true valuations). For each hour of the game, a
broker has to predict the demand and make bids in the wholesale PDA to fulfill
the predicted demand. Each broker can participate in all 24 auctions and can
submit any number of bids. The PowerTAC auction clearing mechanism clears the
market after each periodic auction and communicates the relevant market-
clearing information to each broker. Each broker is informed about the market-
clearing price, its own cleared quantity, and net cleared quantity. The
orderbook contains the anonymized list of uncleared bids and uncleared asks in
the auction. Failing to purchase the required quantity from the wholesale
market makes the broker purchase the remaining quantity from the balancing
market by paying a balancing price that is generally higher than wholesale
prices, which is meant to penalize the broker for creating an imbalance. We
refer the reader to PowerTAC 2020 specifications to know more about the
simulator Ketter (2020).

### Benchmark Strategies

The following are the PDA wholesale strategies used to benchmark DDPGBBS
performance.

  * •

Zero Intelligence (ZI): The ZI strategy follows a randomized approach to bid
in a PDA by ignoring the state of the market; it samples a price from a
uniform distribution between the minimum bid price and maximum bid price.
Following the ZI strategy, the broker places one bid per auction and the
remaining quantity as bid-quantity for all 24 auction instances.

  * •

Zero Intelligence Plus (ZIP): The ZIP agent Tesauro (2001) maintains a scalar
variable $m$ denoting the profit it aims to achieve, which gets combined with
a unit limit price to compute a bid price $p$. Small increments adjust the
price for each trade with the help of a $\delta$ by comparing the submitted
bid price and the clearing price. We decide the bid price $\mu$ randomly at
the start of the game. The profit margin $m$ is set to $-1\%$ of $\mu$,
resulting in the initial bid price to be $p=\mu\ x\ 0.99$.

  * •

VidyutVanika (VV): VidyutVanika Susobhan (2020) agent uses an MDP-based
bidding strategy, where the MDP is solved with the help of dynamic programming
and limit-price predictor. The MDP suggests a suitable limit-price for each of
24 auction instances. Additionally, it uses a heuristic strategy to spread the
bid-quantity across 24 instances based on the predicted limit-prices.

  * •

SPOT: The SPOT Chowdhury (2018) agent uses a Monte Carlo Tree Search (MCTS)
based strategy with coupled heuristics on top of the limit price derived from
a REPTree based limit price predictor to determine the optimal bid price. This
strategy places more than one bid in each auction instance by equally
distributing the remaining required quantity among all the bids.

##  4 Theoretical Analysis

Here, we show the theoretical analysis of double auctions. Consider a two-unit
double auction involving one buyer and one seller, employing ACPR with uniform
pricing as the payment rule. Both buyer and seller follow a scale-based
bidding strategy $b_{B}$ and $b_{S}$, respectively. The buyer places two bids
$b_{B}^{1}=\alpha_{B1}\theta_{B}$ and $b_{B}^{2}=\alpha_{B2}\theta_{B}$, while
the seller places two asks $b_{S}^{1}=\alpha_{S1}\theta_{S}$ and
$b_{S}^{2}=\alpha_{S2}\theta_{S}$. In the next section, we formalize the
expected utilities of the buyer and the seller for a general two-unit double
auction, where both the buyer and seller follow scale-based bidding
strategies, and their types are drawn from $U[l_{B},h_{B}]$ and
$U[l_{S},h_{S}]$, respectively.

###  4.1 Nash Equilibrium Analysis for Multi-unit Double Auctions

We start with an analysis of the buyer’s expected utility. There are three
possible clearing scenarios; We can write separate equations for each of the
three scenarios and later combine them. $u_{B}^{1}$ is the expected utility
when both units are cleared, $u_{B}^{2}$ is the expected utility when only one
unit is cleared, and $u_{B}^{3}$ is the expected utility when no clearing
happens. As the market does not clear in the third scenario, $u_{B}^{3}=0$. We
can write them as follows:

$\displaystyle u_{B}^{1}$
$\displaystyle=2\int_{l_{s}}^{\frac{\alpha_{B2}}{\alpha_{S2}}\theta_{B}}\left[\theta_{B}-\left(\frac{\alpha_{B2}\theta_{B}+\alpha_{S2}\theta_{S}}{2}\right)\right].\left(\frac{1}{h_{S}-l_{S}}\right)\
d\theta_{S}$ (1)

$\displaystyle u_{B}^{2}$
$\displaystyle=\int_{\frac{\alpha_{B2}}{\alpha_{S2}}\theta_{B}}^{\frac{\alpha_{B1}}{\alpha_{S1}}\theta_{B}}\left[\theta_{B}-\left(\frac{\alpha_{B1}\theta_{B}+\alpha_{S1}\theta_{S}}{2}\right)\right].\left(\frac{1}{h_{S}-l_{S}}\right)\
d\theta_{S}$ (2)

Assuming that the buyer decides to fix its $\alpha_{B1}$ and $\alpha_{B2}$
before even seeing its own type,

$\displaystyle U_{B}$
$\displaystyle=\int_{l_{B}}^{h_{B}}\left(u_{B}^{1}+u_{B}^{2}\right).\left(\frac{1}{h_{B}-l_{B}}\right)\
d\theta_{B}$ (3)

Equations 1 and 2 calculate the interim utility of the buyer for the scenarios
involving two units of clearance and one unit clearance, respectively. The
integral limits in Equations 1 and 2 denote the regions where two units of
clearance and one unit clearance happen, respectively. As we assume that the
true types are sampled from the uniform distribution, the normalizing factors
denote the probabilities of sampling from such a distribution. For the last
case, when no clearing happens, the interim utility of the buyer is zero.
Equation 3 calculates the expected utility of the buyer based on the above
three scenarios. Solving Equation 3 would result in,

$\displaystyle U_{B}$
$\displaystyle=\frac{(h_{B}^{2}+h_{B}l_{B}+l_{B}^{2})}{3(h_{S}-l_{S})}\Bigg{[}\Bigg{.}\left(\frac{\alpha_{B1}}{\alpha_{S1}}-\frac{\alpha_{B2}}{\alpha_{S2}}\right)\left(1-\frac{\alpha_{B1}}{2}-\frac{\alpha_{S1}}{4}\left(\frac{\alpha_{B1}}{\alpha_{S1}}+\frac{\alpha_{B2}}{\alpha_{S2}}\right)\right)+2\frac{\alpha_{B2}}{\alpha_{S2}}\left(1-\frac{\alpha_{B2}}{2}\right)-\frac{\alpha_{B2}^{2}}{2\alpha_{S2}}\Bigg{]}\Bigg{.}$
$\displaystyle-
l_{S}\frac{(h_{B}+l_{B})}{(h_{S}-l_{S})}\left(1-\frac{\alpha_{B2}}{2}\right)+\frac{1}{2(h_{S}-l_{S})}l_{S}^{2}\alpha_{S2}$
(4)

We now calculate the seller’s expected utility following the similar logic as
above,

$\displaystyle u_{S}^{1}$
$\displaystyle=2\int_{\frac{\alpha_{S2}\theta_{S}}{\alpha_{B2}}}^{h_{B}}\left[\left(\frac{\alpha_{B2}\theta_{B}+\alpha_{S2}\theta_{S}}{2}\right)-\theta_{S}\right].\left(\frac{1}{h_{B}-l_{B}}\right)\
d\theta_{B}$ (5)

$\displaystyle u_{S}^{2}$
$\displaystyle=\int_{\frac{\alpha_{S1}\theta_{S}}{\alpha_{B1}}}^{\frac{\alpha_{S2}\theta_{S}}{\alpha_{B2}}}\left[\left(\frac{\alpha_{B1}\theta_{B}+\alpha_{S1}\theta_{S}}{2}\right)-\theta_{S}\right].\left(\frac{1}{h_{B}-l_{B}}\right)\
d\theta_{B}$ (6)

Assuming that the seller decides to fix its $\alpha_{S1}$ and $\alpha_{S2}$
before even seeing its own type,

$\displaystyle U_{S}$
$\displaystyle=\int_{l_{S}}^{h_{S}}\left(u_{S}^{1}+u_{S}^{2}\right).\left(\frac{1}{h_{S}-l_{S}}\right)\
d\theta_{S}$ (7)

Solving the above equations would result in,

$\displaystyle U_{S}$
$\displaystyle=\frac{(h_{S}^{2}+h_{S}l_{S}+l_{S}^{2})}{3(h_{B}-l_{B})}\Bigg{[}\Bigg{.}\left(\frac{\alpha_{S2}}{\alpha_{B2}}-\frac{\alpha_{S1}}{\alpha_{B1}}\right)\left(\frac{\alpha_{B1}}{4}\left(\frac{\alpha_{S2}}{\alpha_{B2}}+\frac{\alpha_{S1}}{\alpha_{B1}}\right)+\frac{\alpha_{S1}}{2}-1\right)+2\frac{\alpha_{S2}}{\alpha_{B2}}\left(1-\frac{\alpha_{S2}}{2}\right)-\frac{\alpha_{S2}^{2}}{2\alpha_{B2}}\Bigg{]}\Bigg{.}$
$\displaystyle-
h_{B}\frac{(h_{S}+l_{S})}{(h_{B}-l_{B})}\left(1-\frac{\alpha_{S2}}{2}\right)+\frac{1}{2(h_{B}-l_{B})}h_{B}^{2}\alpha_{B2}$
(8)

Now, We consider the following four possible scenarios based on the scale
factors of the buyer and the seller:
Case-1: Same Scale-Factors for Buyer, Same Scale Factors for Seller
Case-2: Different Scale Factors for Buyer, Same Scale Factors for Seller
Case-3: Same Scale Factors for Buyer, Different Scale Factors for Seller
Case-4: Different Scale Factors for Buyer, Different Scale Factors for Seller

####  4.1.1 Case-1: Same Scale Factors for Buyer, Same Scale Factors for
Seller

Here $\alpha_{B1}=\alpha_{B2}=\alpha_{B}$ and
$\alpha_{S1}=\alpha_{S2}=\alpha_{S}$. Rewriting equations (3) and (6) for this
case,

$\displaystyle U_{B}$
$\displaystyle=\frac{(h_{B}^{2}+h_{B}l_{B}+l_{B}^{2})}{6\alpha_{S}(h_{S}-l_{S})}(\alpha_{B}(4-3\alpha_{B}))-l_{S}\frac{(h_{B}+l_{B})}{(h_{S}-l_{S})}\left(1-\frac{\alpha_{B}}{2}\right)+\frac{1}{2(h_{S}-l_{S})}l_{S}^{2}\alpha_{S}$
(9)

$\displaystyle U_{S}$
$\displaystyle=\frac{(h_{S}^{2}+h_{S}l_{S}+l_{S}^{2})}{6\alpha_{B}(h_{B}-l_{B})}(\alpha_{S}(4-3\alpha_{S}))-h_{B}\frac{(h_{S}+l_{S})}{(h_{B}-l_{B})}\left(1-\frac{\alpha_{S}}{2}\right)+\frac{1}{2(h_{B}-l_{B})}h_{B}^{2}\alpha_{B}$
(10)

Now, in order to find the $\alpha_{B}$ that maximizes the buyer’s expected
utility $U_{B}$, we take $\frac{\partial{U_{B}}}{\partial{\alpha_{B}}}=0$,
which implies

$\displaystyle\frac{(h_{B}^{2}+h_{B}l_{B}+l_{B}^{2})}{6\alpha_{S}(h_{S}-l_{S})}(4-6\alpha_{B})+\frac{l_{S}}{2}\left(\frac{h_{B}+l_{B}}{h_{S}-l_{S}}\right)=0\Rightarrow$
$\displaystyle\frac{(h_{B}^{2}+h_{B}l_{B}+l_{B}^{2})}{(h_{B}+l_{B})}(4-6\alpha_{B})+3l_{S}\alpha_{S}=0\Rightarrow$
$\displaystyle
4-6\alpha_{B}=-3l_{S}\alpha_{S}\left(\frac{h_{B}+l_{B}}{h_{B}^{2}+h_{B}l_{B}+l_{B}^{2}}\right)\Rightarrow$
$\displaystyle\alpha_{B}=\frac{2}{3}+\frac{l_{S}\alpha_{S}}{2}\left(\frac{h_{B}+l_{B}}{h_{B}^{2}+h_{B}l_{B}+l_{B}^{2}}\right)$
(11)

To find the $\alpha_{S}$ that maximizes the seller’s expected utility $U_{S}$,
we take $\frac{\partial{U_{S}}}{\partial{\alpha_{S}}}=0$, which implies

$\displaystyle\frac{(h_{S}^{2}+h_{S}l_{S}+l_{S}^{2})}{6\alpha_{B}(h_{B}-l_{B})}(4-6\alpha_{S})+\frac{h_{B}}{2}\frac{(h_{S}+l_{S})}{(h_{B}-l_{B})}=0\Rightarrow$
$\displaystyle\frac{(h_{S}^{2}+h_{S}l_{S}+l_{S}^{2})}{(h_{S}+l_{S})}(4-6\alpha_{S})+3h_{B}\alpha_{B}=0\Rightarrow$
$\displaystyle
4-6\alpha_{S}=-3h_{B}\alpha_{B}\left(\frac{h_{S}+l_{S}}{h_{S}^{2}+h_{S}l_{S}+l_{S}^{2}}\right)\Rightarrow$
$\displaystyle\alpha_{S}=\frac{2}{3}+\frac{h_{B}\alpha_{B}}{2}\left(\frac{h_{S}+l_{S}}{h_{S}^{2}+h_{S}l_{S}+l_{S}^{2}}\right)$
(12)

Above $\alpha_{B}$ and $\alpha_{S}$ calculate scale factors for a generic case
when $\theta_{B}\sim U[l_{B},h_{B}]$ and $\theta_{S}\sim U[l_{S},h_{S}]$.

  * Remark

For a specific case of $\theta_{B}\sim U[0,1]$ and $\theta_{S}\sim U[0,1]$,
replacing $l_{B}=l_{S}=0$ and $h_{S}=h_{B}=1$ in the above equations would
result in $\alpha_{B}=\frac{2}{3}$ and $\alpha_{S}=1$.

####  4.1.2 Case-2: Different Scale Factors for Buyer, Same Scale Factors for
Seller

Here $\alpha_{S1}=\alpha_{S2}=\alpha_{S}$. Rewriting equations (3) and (6) for
this case,

$\displaystyle U_{B}$
$\displaystyle=\frac{(h_{B}^{2}+h_{B}l_{B}+l_{B}^{2})}{3\alpha_{S}(h_{S}-l_{S})}\Bigg{(}\Bigg{.}(\alpha_{B1}-\alpha_{B2})\left(1-\frac{3\alpha_{B1}}{4}-\frac{\alpha_{B2}}{4}\right)-\frac{3\alpha_{B2}^{2}}{2}+2{\alpha_{B2}}\Bigg{)}\Bigg{.}-l_{S}\frac{(h_{B}+l_{B})}{(h_{S}-l_{S})}\left(1-\frac{\alpha_{B2}}{2}\right)$
$\displaystyle+\frac{1}{2(h_{S}-l_{S})}l_{S}^{2}\alpha_{S}$ (13)
$\displaystyle U_{S}$
$\displaystyle=\frac{(h_{S}^{2}+h_{S}l_{S}+l_{S}^{2})}{3(h_{B}-l_{B})}\Bigg{(}\Bigg{.}\alpha_{S}\left(\frac{1}{\alpha_{B2}}-\frac{1}{\alpha_{B1}}\right)\left(\frac{\alpha_{S}}{4}\left(\frac{\alpha_{B1}}{\alpha_{B2}}+1\right)+\frac{\alpha_{S}}{2}-1\right)+\frac{\alpha_{S}}{\alpha_{B2}}\left(2-\frac{3\alpha_{S}}{2}\right)\Bigg{)}\Bigg{.}$
$\displaystyle-
h_{B}\frac{(h_{S}+l_{S})}{(h_{B}-l_{B})}\left(1-\frac{\alpha_{S2}}{2}\right)+\frac{1}{2(h_{B}-l_{B})}h_{B}^{2}\alpha_{B2}$
(14)

Now, to find the $\alpha_{B1}$ that maximizes the buyer’s expected utility
$U_{B}$, we take $\frac{\partial{U_{B}}}{\partial{\alpha_{B1}}}=0$, which
implies,

$\displaystyle(\alpha_{B1}-\alpha_{B2})(\frac{-3}{4})+\left(1-\frac{3\alpha_{B1}}{4}-\frac{\alpha_{B2}}{4}\right)=0\Rightarrow$
$\displaystyle 3\alpha_{B1}-\alpha_{B2}-2=0$ (15)

To find the $\alpha_{B2}$ that maximizes $U_{B}$, we take
$\frac{\partial{U_{B}}}{\partial{\alpha_{B2}}}=0$, which implies,

$\displaystyle\frac{(h_{B}^{2}+h_{B}l_{B}+l_{B}^{2})}{3\alpha_{S}(h_{S}-l_{S})}\Bigg{(}\Bigg{.}(\alpha_{B1}-\alpha_{B2})\left(\frac{-1}{4}\right)-\left(1-\frac{3\alpha_{B1}}{4}-\frac{\alpha_{B2}}{4}\right)-3\alpha_{B2}+2\Bigg{)}\Bigg{.}-l_{S}\frac{(h_{B}+l_{B})}{(h_{S}-l_{S})}\left(\frac{-1}{2}\right)=0\Rightarrow$
$\displaystyle(\alpha_{B1}-5\alpha_{B2}+2)\left(\frac{h_{B}^{2}+h_{B}l_{B}+l_{B}^{2}}{h_{B}+l_{B}}\right)+3l_{s}\alpha_{S}=0$
(16)

To find the $\alpha_{S}$ that maximizes $U_{S}$, we take
$\frac{\partial{U_{S}}}{\partial{\alpha_{S}}}=0$, which implies

$\displaystyle\frac{(h_{S}^{2}+h_{S}l_{S}+l_{S}^{2})}{3(h_{B}-l_{B})}\Bigg{[}\Bigg{.}\alpha_{S}\left(\frac{1}{\alpha_{B2}}-\frac{1}{\alpha_{B1}}\right)\left(\frac{1}{4}\left(\frac{\alpha_{B1}}{\alpha_{B2}}+1\right)+\frac{1}{2}\right)+\left(\frac{1}{\alpha_{B2}}-\frac{1}{\alpha_{B1}}\right)\left(\frac{\alpha_{S}}{4}\left(\frac{\alpha_{B1}}{\alpha_{B2}}+1\right)+\frac{\alpha_{S}}{2}-1\right)$
$\displaystyle+\frac{1}{\alpha_{B2}}\left(2-3\alpha_{S}\right)\Bigg{]}\Bigg{.}-h_{B}\frac{(h_{S}+l_{S})}{(h_{B}-l_{B})}\left(\frac{-1}{2}\right)=0\Rightarrow$
$\displaystyle\Bigg{(}\Bigg{.}\Bigg{(}\Bigg{.}\frac{1}{\alpha_{B2}}-\frac{1}{\alpha_{B1}}\Bigg{)}\Bigg{.}(\alpha_{S}(\alpha_{B1}+3\alpha_{B2})-2\alpha_{B2})-6\alpha_{S}+4\Bigg{)}\Bigg{.}\left(\frac{h_{S}^{2}+h_{S}l_{S}+l_{S}^{2}}{h_{S}+l_{S}}\right)+3h_{B}\alpha_{B2}=0$
(17)

  * Remark

For a specific case of $\theta_{B}\sim U[0,1]$ and $\theta_{S}\sim U[0,1]$,
replacing $l_{B}=l_{S}=0$ and $h_{S}=h_{B}=1$ in the above equations, we get
the following set of equations,

$\displaystyle 3\alpha_{B1}-\alpha_{B2}-2=0$ (18)
$\displaystyle\alpha_{B1}-5\alpha_{B2}+2=0$ (19)
$\displaystyle\Bigg{(}\Bigg{.}\frac{1}{\alpha_{B2}}-\frac{1}{\alpha_{B1}}\Bigg{)}\Bigg{.}(\alpha_{S}(\alpha_{B1}+3\alpha_{B2})-2\alpha_{B2})-6\alpha_{S}+4+3\alpha_{B2}=0$
(20)

By solving these equations, we get $\alpha_{B1}=\frac{6}{7}$,
$\alpha_{B2}=\frac{4}{7}$ and $\alpha_{S}\approx 1.12169312$.

####  4.1.3 Case-3: Same Scale Factors for Buyer, Different Scale Factors for
Seller

Here $\alpha_{B1}=\alpha_{B2}=\alpha_{B}$. Rewriting equations (3) and (6) for
this case,

$\displaystyle
U_{B}=\frac{(h_{B}^{2}+h_{B}l_{B}+l_{B}^{2})}{3(h_{S}-l_{S})}\Bigg{[}\Bigg{.}\alpha_{B}\left(\frac{1}{\alpha_{S1}}-\frac{1}{\alpha_{S2}}\right)\left(1-\frac{\alpha_{B}}{2}-\frac{\alpha_{B}}{4}\left(1+\frac{\alpha_{S1}}{\alpha_{S2}}\right)\right)+\frac{\alpha_{B}}{\alpha_{S2}}\left(2-\frac{3\alpha_{B}}{2}\right)\Bigg{]}\Bigg{.}$
$\displaystyle-
l_{S}\frac{(h_{B}+l_{B})}{(h_{S}-l_{S})}\left(1-\frac{\alpha_{B}}{2}\right)+\frac{1}{2(h_{S}-l_{S})}l_{S}^{2}\alpha_{S2}$
(21) $\displaystyle
U_{S}=\frac{(h_{S}^{2}+h_{S}l_{S}+l_{S}^{2})}{3\alpha_{B}(h_{B}-l_{B})}\Bigg{[}\Bigg{.}(\alpha_{S2}-\alpha_{S1})\left(\frac{3\alpha_{S1}}{4}+\frac{\alpha_{S2}}{4}-1\right)-\frac{3\alpha_{S2}^{2}}{2}+2\alpha_{S2}\Bigg{]}\Bigg{.}-h_{B}\frac{(h_{S}+l_{S})}{(h_{B}-l_{B})}\left(1-\frac{\alpha_{S2}}{2}\right)$
$\displaystyle+\frac{1}{2(h_{B}-l_{B})}h_{B}^{2}\alpha_{B}$ (22)

Now, to find the $\alpha_{B1}$ that maximizes the buyer’s expected utility
$U_{B}$, we take $\frac{\partial{U_{B}}}{\partial{\alpha_{B}}}=0$, which
implies,

$\displaystyle\frac{(h_{B}^{2}+h_{B}l_{B}+l_{B}^{2})}{3(h_{S}-l_{S})}\Bigg{[}\Bigg{.}\left(\frac{1}{\alpha_{S1}}-\frac{1}{\alpha_{S2}}\right)\left(1-\alpha_{B}-\frac{\alpha_{B}}{2}\left(1+\frac{\alpha_{S1}}{\alpha_{S2}}\right)\right)+\frac{1}{\alpha_{S2}}\left(2-3\alpha_{B}\right)\Bigg{]}\Bigg{.}-l_{S}\frac{(h_{B}+l_{B})}{(h_{S}-l_{S})}\left(\frac{-1}{2}\right)\Rightarrow$
$\displaystyle\Bigg{(}\Bigg{.}\Bigg{(}\Bigg{.}\frac{1}{\alpha_{S2}}-\frac{1}{\alpha_{S1}}\Bigg{)}\Bigg{.}(\alpha_{B}(\alpha_{S1}+3\alpha_{S2})-2\alpha_{S2})-6\alpha_{B}+4\Bigg{)}\Bigg{.}\left(\frac{h_{B}^{2}+h_{B}l_{B}+l_{B}^{2}}{h_{B}+l_{B}}\right)+3l_{S}\alpha_{S2}=0$
(23)

To find the $\alpha_{B2}$ that maximizes $U_{B}$, we take
$\frac{\partial{U_{S}}}{\partial{\alpha_{S1}}}=0$, which implies,

$\displaystyle\frac{(h_{S}^{2}+h_{S}l_{S}+l_{S}^{2})}{3\alpha_{B}(h_{B}-l_{B})}\Bigg{[}\Bigg{.}(-1)\left(\frac{3\alpha_{S1}}{4}+\frac{\alpha_{S2}}{4}-1\right)+(\alpha_{S2}-\alpha_{S1})\left(\frac{3}{4}\right)\Bigg{]}\Bigg{.}=0\Rightarrow$
$\displaystyle 3\alpha_{S1}-\alpha_{S2}-2=0$ (24)

To find the $\alpha_{S}$ that maximizes $U_{S}$, we take
$\frac{\partial{U_{S}}}{\partial{\alpha_{S}2}}=0$, which implies

$\displaystyle\frac{(h_{S}^{2}+h_{S}l_{S}+l_{S}^{2})}{3\alpha_{B}(h_{B}-l_{B})}\Bigg{[}\Bigg{.}\left(\frac{3\alpha_{S1}}{4}+\frac{\alpha_{S2}}{4}-1\right)+(\alpha_{S2}-\alpha_{S1})\left(\frac{1}{4}\right)-3\alpha_{S2}+2\Bigg{]}\Bigg{.}-h_{B}\frac{(h_{S}+l_{S})}{(h_{B}-l_{B})}\left(-\frac{1}{2}\right)\Rightarrow$
$\displaystyle(\alpha_{S1}-5\alpha_{S2}+2)\left(\frac{h_{S}^{2}+h_{S}l_{S}+l_{S}^{2}}{h_{S}+l_{S}}\right)+3h_{B}\alpha_{B}=0$
(25)

  * Remark

For a specific case of $\theta_{B}\sim U[0,1]$ and $\theta_{S}\sim U[0,1]$,
replacing $l_{B}=l_{S}=0$ and $h_{S}=h_{B}=1$ in the above equations, we get
the following set of equations,

$\displaystyle\Bigg{(}\Bigg{.}\frac{1}{\alpha_{S2}}-\frac{1}{\alpha_{S1}}\Bigg{)}\Bigg{.}(\alpha_{B}(\alpha_{S1}+3\alpha_{S2})-2\alpha_{S2})-6\alpha_{B}+4=0$
(26) $\displaystyle 3\alpha_{S1}-\alpha_{S2}-2=0$ (27)
$\displaystyle\alpha_{S1}-5\alpha_{S2}+2+3\alpha_{B}=0$ (28)

By solving these equations, we get $\alpha_{B}=\frac{2}{3}$, $\alpha_{S1}=1$
and $\alpha_{S2}=1$.

####  4.1.4 Case-4: Different Scale Factors for Buyer, Different Scale Factors
for Seller

  * Remark

If we perform the similar analysis using Equations 4 and 8, we get the
following equations,

$\displaystyle\alpha_{S1}\alpha_{B2}+\alpha_{S2}(2-3\alpha_{B1})=0$ (29)
$\displaystyle\Bigg{(}\Bigg{.}2+\alpha_{B1}-6\alpha_{B2}+\frac{\alpha_{S1}\alpha_{B2}}{\alpha_{S2}}\Bigg{)}\Bigg{.}\left(\frac{h_{B}^{2}+h_{B}l_{B}+l_{B}^{2}}{h_{B}+l_{B}}\right)+3l_{S}\alpha_{S2}=0$
(30) $\displaystyle\alpha_{B1}\alpha_{S2}+\alpha_{B2}(2-3\alpha_{S1})=0$ (31)
$\displaystyle\Bigg{(}\Bigg{.}2+\alpha_{S1}-6\alpha_{S2}+\frac{\alpha_{B1}\alpha_{S2}}{\alpha_{B2}}\Bigg{)}\Bigg{.}\left(\frac{h_{S}^{2}+h_{S}l_{S}+l_{S}^{2}}{h_{S}+l_{S}}\right)+3h_{B}\alpha_{B2}=0$
(32)

  * Remark

For a specific case of $\theta_{B}\sim U[0,1]$ and $\theta_{S}\sim U[0,1]$,
replacing $l_{B}=l_{S}=0$ and $h_{S}=h_{B}=1$ in the above equations, we get
the following set of equations,

$\displaystyle\alpha_{S1}\alpha_{B2}+\alpha_{S2}(2-3\alpha_{B1})=0$ (33)
$\displaystyle\alpha_{S2}(2+\alpha_{B1}-6\alpha_{B2})+\alpha_{S1}\alpha_{B2}=0$
(34) $\displaystyle\alpha_{B1}\alpha_{S2}+\alpha_{B2}(2-3\alpha_{S1})=0$ (35)
$\displaystyle\alpha_{B2}(2+\alpha_{S1}-6\alpha_{S2})+\alpha_{B1}\alpha_{S2}+3\alpha_{B2}^{2}=0$
(36)

By solving these equations, we get $\alpha_{B1}\approx 0.882782$,
$\alpha_{B2}\approx 0.588521$, $\alpha_{S1}\approx 1.2207$ and
$\alpha_{S2}\approx 1.10806$.

We summarise the above discussion as the following theorem,

######  Theorem 4.1.

For a two-unit k-double auction with k = 0.5, where one buyer and one seller
participate in the auction; the buyer’s and the seller’s true types are drawn
from $U_{B}\leavevmode\nobreak\ [l_{B},h_{B}]$ and $U_{S}\leavevmode\nobreak\
[l_{S},h_{S}]$, respectively; when they deploy scale based bidding strategies
$b_{B}$ and $b_{S}$ and fix their scaling factors before even seeing their
true types, below set of equations characterise the equilibrium,

  * •

If $\alpha_{B1}=\alpha_{B2}=\alpha_{B}$ and
$\alpha_{S1}=\alpha_{S2}=\alpha_{S}$, then Equation 11 and 12 characterise
BNE.

  * •

If $\alpha_{B1}\neq\alpha_{B2}$ and $\alpha_{S1}=\alpha_{S2}=\alpha_{S}$, then
Equation 15, 16 and 17 characterise BNE.

  * •

If $\alpha_{B1}=\alpha_{B2}=\alpha_{B}$ and $\alpha_{S1}\neq\alpha_{S2}$, then
Equation 23, 24 and 25 characterise BNE.

  * •

If $\alpha_{B1}\neq\alpha_{B2}$ and $\alpha_{S1}\neq\alpha_{S2}$, then
Equation 29, 30, 31 and 32 characterise BNE.

######  Corollary 4.2.

For a two-unit k-double auction with k = 0.5, where one buyer and one seller
participate in the auction; the buyer’s and the seller’s true types are drawn
from $U\leavevmode\nobreak\ [0,1]$, when they deploy scale based bidding
strategies $b_{B}$ and $b_{S}$ and fix their scaling factors before even
seeing their true types, below scale factors constitute the BNE for each case,

  * •

If $\alpha_{B1}=\alpha_{B2}=\alpha_{B}$ and
$\alpha_{S1}=\alpha_{S2}=\alpha_{S}$, then $\alpha_{B}=\frac{2}{3}$ and
$\alpha_{S}=1$ constitute Nash Equilibrium.

  * •

If $\alpha_{B1}\neq\alpha_{B2}$ and $\alpha_{S1}=\alpha_{S2}=\alpha_{S}$, then
$\alpha_{B1}=\frac{6}{7}$, $\alpha_{B2}=\frac{4}{7}$ and $\alpha_{S}\approx
1.12169312$ constitute Nash Equilibrium.

  * •

If $\alpha_{B1}=\alpha_{B2}=\alpha_{B}$ and $\alpha_{S1}\neq\alpha_{S2}$, then
$\alpha_{B}=\frac{2}{3}$, $\alpha_{S1}=1$ and $\alpha_{S2}=1$ constitute Nash
Equilibrium.

  * •

If $\alpha_{B1}\neq\alpha_{B2}$ and $\alpha_{S1}\neq\alpha_{S2}$, then
$\alpha_{B1}\approx 0.882782$, $\alpha_{B2}\approx 0.588521$,
$\alpha_{S1}\approx 1.2207$ and $\alpha_{S2}\approx 1.10806$ constitute Nash
Equilibrium.

##  5 Building DDPG based Bidding Strategy

In this section, we start with an overview of DDPG and then present arguments
that motivated us to construct a bidding strategy using DDPG. We, then,
describe DDPGBBS in detail along with the training and validation set-up for
this novel bidding strategy. Additionally, we furnish experimental results
that validate that DDPGBBS indeed converges to the theoretical equilibrium.
Finally, we extend our DDPGBBS implementation intending to utilize it in
smart-grids.

###  5.1 Deep Deterministic Policy Gradient (DDPG)

Deep Deterministic Policy Gradient Lillicrap (2016), commonly known as DDPG,
is an actor-critic, model-free reinforcement learning (RL) algorithm based on
the concept of deterministic policy gradients Silver (2014). This algorithm
works on continuous action spaces and has been proved to be very successful in
a variety of continuous control problems Lillicrap (2016), e.g., TORCS and
MUJOCO environments. The algorithm deploys a parameterized policy (actor) and
critic network that gets updated frequently using experiences gathered from
sample roll-outs. While the policy network map learns an optimal deterministic
mapping from state space to action space, the critic network learns the
optimal value function using Bellman type updates used in Q-learning. Our
choice of using the DDPG framework to solve the optimal bidding problem is due
to our requirement that the bidding decisions of the proposed strategy are
continuous in nature. Having motivated our choice of RL algorithm, we now
proceed to describe the formulation of the bidding problem using the Markov
decision process Puterman (1994). Figure 1 shows the structural framework of
the DDPG algorithm.

Figure 1: The DDPG Algorithm Structure Framework Kang (2021)

###  5.2 DDPGBBS: A Bidding Strategy for Multi-unit Auctions

In this section, we present details of our DDPGBBS based formulation. Here,
the goal is to learn the buyer’s Nash equilibrium bidding strategy, i.e.,
scale-factors of buyer for each of the possible scenarios discussed earlier.
We propose the following MDP framework,

The state space $S$ consists of _quantity to buy_ $q$, where $q\in Q$,
$Q=\\{0,1,2\\}$; and _Buyer’s true type_ $\theta\in[0,1]$. The actions are the
buyer’s scale-factors $\alpha_{B1}$, $\alpha_{B2}\in[0,1]$, which want DDPGBBS
to learn. Buyer receives reward $r=0$ if no market-clearing happens, else it
receives reward $r=-cp*cq$ (where $cp$ and $cq$ are clearing price and buyer’s
clearing quantity). As we are working with negative rewards, an optimal
strategy would be one that maximizes the expected reward. A state $s$
transition to the next state where _quantity to buy_ is the remaining quantity
$q^{\prime}$ after auction-clearing in state $s$, ($q^{\prime}\in
Q=\\{0,1,2\\}$), while _buyer’s true type_ remains the same. Here we are
considering single-shot auction, thus MDP terminates after a single iteration.
Since, there is no possible action from the terminal state, a buyer will
receive a terminal reward $r=-q^{\prime}*\theta$

The DDPG architecture details are as follows: the neural network used for
actor-network had two hidden layers having 40 and 30 units, respectively. The
critic networks also had two hidden layers with 40 and 30 units, respectively.
Actions were not included until the second hidden layer of the critic network.
Both networks used the _ReLU_ activation for all hidden layers. The actor-
network used _sigmoid_ , while the critic network used the _linear_ activation
in the output layer. The actor-network had two units in the input layer and
two units in the output layer. We used _Adam optimizer_ for learning the
neural network parameters and kept the learning rate of $10^{-4}$ and
$10^{-3}$ for the actor and critic, respectively. we used a discount factor of
$\gamma=0.99$. For the soft target updates, we used $\tau=0.001$.

###  5.3 Training and Validation Set-up of DDPGBBS

In order to train our DDPGBBS, we designed a simulation of a multi-unit
$k$-double auction, which includes the proposed scale-based bidding strategy
of the buyer and the seller. Here, we aim to train DDPGBBS to learn the
theoretical equilibrium of the buyer, but the same setup can be used to learn
the seller’s theoretical equilibrium as well. As shown in Section 4, there are
four possible scenarios based on scale factors of the buyer and seller. But,
Case-1 and Case-3 lead to the same equilibrium; thus we consider them as a
single scenario.

Now, we train a separate DDPGBBS to learn the buyer’s equilibrium for each of
the three cases. We model the seller to play its scale-based equilibrium
bidding strategy by fixing the scale factor according to theoretical results
for each case, while its type is drawn from $U\sim[0,1]$. Then, we make our
buyer select scale factors uniformly at random from the interval $[0,1]$ and
select its type from $U\sim[0,1]$ and prepare bids using the scale-based
bidding strategy. The simulated auction clearing mechanism clears the auction
and decides the clearing price and quantity. After each auction clearing, the
buyer receives a reward and information about the next state, and the episode
terminates. A buyer prepares an experience quadruple
$(s_{t},a_{t},r_{t},s_{t+1})$ for each auction episode and stores it to a
replay buffer. After sampling $r$ independent experience samples from replay
buffer, we update the policy and critic networks. We stop training after $n$
number of iterations; In our experiments we kept $r=100000$ and $n=10000$.

###  5.4 Validating Theoretical Results

Here we discuss the results of the validation experiments. As discussed above,
there are three possible scenarios and we trained a separate DDPGBBS model for
each of the three cases. Table 1 compares buyer’s equilibrium scale factors
learned by DDPGBBS against the theoretical scale factors for the case when
$\alpha_{B1}=\alpha_{B2}=\alpha_{B}$ and $\alpha_{S1}=\alpha_{S2}=\alpha_{S}$.
Similarly, Table 2 compares buyer’s equilibrium scale factors learned by
DDPGBBS against the theoretical scale factors for the case when
$\alpha_{B1}\neq\alpha_{B2}$ and $\alpha_{S1}=\alpha_{S2}=\alpha_{S}$.
Finally, Table 3 compares buyer’s equilibrium scale factors learned by DDPGBBS
against the theoretical scale factors for the general case when
$\alpha_{B1}\neq\alpha_{B2}$ and $\alpha_{S1}\neq\alpha_{S2}$. Note that the
buyer following DDPGBBS does not get to see seller’s asks (neither during
training nor during validation), and only learns the equilibrium strategy
based on the feedback it receives from the auction clearing mechanism for its
own actions. These results demonstrate that DDPGBBS is able to detect the
choice of seller’s scale factors and effectively learn the Nash equilibrium
strategy for each case.

Table 1: Experimental scale factor values of buyer compared with theoretical scale factor values for case-1 |  | $\alpha_{B1}$
---|---|---
Theoretical Equilibrium |  | 0.666667
DDPG Empirical Equilibrium | mean | 0.577733
| std | 0.023132
Table 2: Experimental scale factor values of buyer compared with theoretical scale factor values for case-2 |  | $\alpha_{B1}$ | $\alpha_{B2}$
---|---|---|---
Theoretical Equilibrium |  | 0.857143 | 0.571428
DDPG Empirical Equilibrium | mean | 0.928814 | 0.477797
| std | 0.022228 | 0.046001
Table 3: Experimental scale factor values of buyer compared with theoretical scale factor values for case-3 |  | $\alpha_{B1}$ | $\alpha_{B2}$
---|---|---|---
Theoretical Equilibrium |  | 0.882782 | 0.588521
DDPG Empirical Equilibrium | mean | 0.855816 | 0.302352
| std | 0.050594 | 0.221870

From Table 1, we can see that the empirical result we obtained using DDPGBBS
is within $12.2\%$ of theoretical $\alpha_{B1}$. From Table 2 (3), learned
$\alpha_{B1}$ and $\alpha_{B2}$ are within $8.28\%(3.13\%)$ and
$16.31\%(47.6\%)$ of the theoretical values, respectively. As we can see,
except for $\alpha_{B2}$ in Table 3, DDPGBBS results are reasonably close to
the theoretical results. As a comparison, the results presented by Susobhan et
al. (in Table 1) Susobhan (2020) for single-unit k-double auction showed
$20.19\%$ and $24.72\%$ difference between empirical results and theoretical
values for buyer and seller, respectively. Additionally, DDPGBBS showed low
variances for all the scale factors except $\alpha_{B2}$ in Table 3,
reinforcing our DDPGBBS’s stability.

###  5.5 Extending DDPGBBS to Design Bidding Strategies for Smart-grids

Here, we extend DDPGBBS to use in general PDA, which we call _Extended
DDPGBBS_. As PDA allows multiple auction instances for a delivery slot, our
DDPGBBS needs to incorporate that information in the state-space. Below is the
updated MDP to use in PDA,

The state space $S$ consists of _proximity_ $p$, where $p\in P$,
$P={0,1,2,...,24}$; _quantity to buy_ $q$, where $q\in R$; and _Buyer’s true
type_ $\theta\in[0,1]$. Here, we consider the buyer’s true type as _average
unit balancing price for buying_ from balancing market in a game. The actions
are the buyer’s scale-factors $\alpha_{B1}$, $\alpha_{B2}\in[0,1]$. During the
game, Extended DDPGBBS outputs these two scale-factors which gets multiplied
with buyer’s true valuation to form the two bids in the auction, while
required bidding quantity is equally distributed into these two bids.

The reward depends on the current state $s$. If $s$ is not a terminal state,
our buyer receives reward $r=0$ if no market-clearing happens, else it
receives reward $r=-cp*cq$ (where $cp$ and $cq$ are clearing price and buyer’s
clearing quantity); if $s$ is a terminal state then it receives reward
$r=-q*\theta$, where $q$ is the remaining quantity at the end of all $24$
auctions. Again, as we are working with negative rewards, an optimal strategy
would be the one that minimizes the expected reward. After each auction, state
transition takes place, and MDP moves to the next state. _Proximity_ changes
$p$ to $p-1$, _quantity to buy_ becomes the remaining quantity after current
auction-clearing, and _buyer’s true type_ remains the same. The episode ends
when the buyer reaches the terminal state $T$, the buyer moves to $T$ either
when Proximity becomes zero or when the buyer buys all the required quantity.

We train this Extended DDPGBBS in offline fashion by collecting experiences in
the replay buffer, using the PowerTAC PDA simulator. To collect experiences,
we run two sets of experiments, with 20 games in each set. We make the
Extended DDPGBBS to play against a ZI broker in two-player games in the first
set. In the second set, Extended DDPGBBS competes against three other ZI
brokers in four-player games. In each set, we distribute hourly demand equally
between all the competing brokers; as a result, each broker had to participate
equally in the wholesale market PDA. Extended DDPGBBS updates the replay
buffer after each auction instance in the game. We trained our Extended
DDPGBBS against ZI brokers as they do not follow any particular bidding
pattern, and thus Extended DDPGBBS gets to see a wide range of states in the
state-space $S$, which improves its learning. After the execution of both sets
is completed, we update our Extended DDPGBBS using the combined replay buffer
of both sets by following the standard DDPG update procedure used for
validation experiments.

##  6 Experiments and Results

Here, we analyze the efficacy of Extended DDPGBBS in PDA by leveraging the
PowerTAC PDA simulator. We first explain the experimental setup followed by
the results of the offline experiments.

To compare the performance of Extended DDPGBBS we isolate the PowerTAC
wholesale market while keeping the default market participants. We benchmark
the performance of Extended DDPGBBS against the state-of-the-art and baseline
strategies mentioned in Section 3.5.

We perform two batches of experiments; the first batch of experiments are
divided into four sets. In each of these four sets, we play ten two-player
games where Extended DDPGBBS is one of the two brokers in each set, while the
second broker is selected from the list of four benchmarking brokers. In other
words, Extended DDPGBBS competes against each benchmarking broker in a set of
10 two-player games. Similarly, in the second batch, we play ten five-player
games having all the available brokers in the game. We chose two-player and
five-player game configurations as the PowerTAC2020 tournament evaluated the
participating brokers on the same game configurations; they also had eight-
player configurations, but the unavailability of that many baseline wholesale
strategies restricted us from performing experiments for the eight-player game
configuration.

Table 4 summarises the results of the first batch of experiments; it shows the
normalized average unit clearing price of each set with respect to the
Extended DDPGBBS’s clearing price. A value greater than $1$ indicates that the
competing broker in that set had a higher average clearing price than Extended
DDPGBBS after playing ten games. Based on the results in Table 4, we observe
that Extended DDPGBBS outperforms all the other bidding strategies
consistently by at least $9.9\%$ in two-player games while achieving 33.76%
improvement against ZI. Figure 2 shows the result for the second batch of
experiments, where we compare the average unit clearing price of each broker
across ten games in a five-player game configuration. The error bars on top of
bar plots show the standard deviation of average clearing prices in 10 games.
Here too, Extended DDPGBBS consistently outperforms all the other brokers by
21.42% against second performing VV, while achieving almost 46% improvement
against other brokers. DDPGBBS obtained better results than the state-of-the-
art strategies (VV and SPOT) used for comparison. DDPGBBS improves in the
range $[21.42\%,46\%]$ 2 over the baseline strategies as well as over VV and
SPOT. Susobhan et al.’s VV (in Figure 1 Susobhan (2020)) improved in the range
$[-11.46\%,22.01\%]$, while Chowdhury et al. (in Figure 3 in Chowdhury (2018))
improved in the range $[31.28\%,32.7\%]$ over previous strategies. It is also
to be noted that, unlike some of the baseline brokers, Extended DDPGBBS does
not incorporate any additional heuristics in its bidding strategy and still
outperforms some of the best brokers of PowerTAC.

Figure 2: Average Unit Clearing Price Comparison for five-player game configuration Table 4: Normalized unit clearing price comparison w.r.t. Extended DDPGBBS SPOT | VV | ZI | ZIP
---|---|---|---
1.2832 | 1.0992 | 1.3376 | 1.1920

##  7 Conclusion

In this paper, we first characterized the Bayesian Nash equilibrium for
single-buyer single-seller multi-unit single-shot k-Double auctions, where the
buyer and the seller follow scale-based bidding strategies. We analyzed all
possible cases resulting from how the buyer and the seller select their scale-
factors in Theorem 4.1. As the equilibrium analysis becomes intractable with
the increase in the number of participating players and items, we presented a
deep deterministic policy gradient bidding strategy DDPGBBS that is easily
extendable to a real-world PDA. We experimentally validated that DDPGBBS
achieves BNE approximately for each case. Furthermore, we proposed an
extension of DDPGBBS that can be utilized in the real-world PDA. Finally, to
examine the Extended DDPGBBS efficacy, we benchmarked it against several
baselines and the state-of-the-art bidding strategy of PowerTAC wholesale PDA.
We showed that it consistently outperforms some of the best PowerTAC wholesale
bidding strategies.

## References

  * Auction-Wikipedia [2021] Auction-Wikipedia.  https://https://en.wikipedia.org/wiki/Auction, 2021.  [Online; accessed 8-October-2021].
  * Y Narahari [2014] Y Narahari.  Game Theory and Mechanism Design, volume 4, chapter 20, pages 303–307.  World Scientific Publishing Co. Pte. Ltd., 2014.
  * Safak [2014] I. Safak Bayram, Muhammad Z. Shakir, Mohamed Abdallah, and Khalid Qaraqe.  A survey on energy trading in smart grid.  In 2014 IEEE Global Conference on Signal and Information Processing (GlobalSIP), pages 258–262, 2014.
  * Nord Pool [2020] Nord Pool AS.  Anual Report.  https://www.nordpoolgroup.com/49eea7/globalassets/download-center/annual-report/annual-review-2020.pdf, 2020\.  [Online; accessed 6-October-2021].
  * Vetsikas [2008] Ioannis Vetsikas and Nicholas Jennings.  Bidding strategies for realistic multi-unit sealed-bid auctions.  Autonomous Agents and Multi-Agent Systems, 21, 01 2008.
  * Shi [2010] Bing Shi, Enrico Gerding, Perukrishnen Vytelingum, and Nick Jennings.  An equilibrium analysis of competing double auction marketplaces using fictitious play.  In 19th European Conference on Artificial Intelligence (ECAI) (01/08/10), pages 575–580, August 2010.
  * Susobhan [2020] Susobhan Ghosh, Sujit Gujar, Praveen Paruchuri, Easwar Subramanian, and Sanjay Bhat.  Bidding in smart grid pdas: Theory, analysis and strategy.  Proceedings of the AAAI Conference on Artificial Intelligence, 34(02):1974–1981, Apr. 2020.
  * Ketter [2020] Wolfgang Ketter, John Collins, and Mathijs de Weerdt.  The 2020 power trading agent competition.  In SSRN Electronic Journal, 2020.
  * Urieli [2014] Daniel Urieli and Peter Stone.  Tactex’13: A champion adaptive power trading agent.  In Association for the Advancement of Artificial Intelligence (AAAI), 2014.
  * Chowdhury [2018] Moinul Morshed Porag Chowdhury, Christopher Kiekintveld, Son Tran, and William Yeoh.  Bidding in periodic double auctions using heuristics and dynamic monte carlo tree search.  In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, pages 166–172. International Joint Conferences on Artificial Intelligence Organization, 7 2018\.
  * Lillicrap [2016] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra.  Continuous control with deep reinforcement learning.  In International Conference on Learning Representations (ICLR), 2016\.
  * Chatterjee [1983] Kalyan Chatterjee and William Samuelson.  Bargaining under incomplete information.  In Operations Research, volume 31, 1983.
  * Satterthwaite [1989] Mark A Satterthwaite and Steven R Williams.  Bilateral trade with the sealed bid k-double: Existence and efficiency.  In Journal of Economic Theory, volume 48, pages 107–133. Multidisciplinary Digital Publishing Institute, 1989.
  * Vetsikas [2014] Ioannis A. Vetsikas.  Equilibrium strategies for multi-unit sealed-bid auctions with multi-unit demand bidders.  In Proceedings of the 2014 International Conference on Autonomous Agents and Multi-Agent Systems, AAMAS ’14, page 1053–1060, Richland, SC, 2014. International Foundation for Autonomous Agents and Multiagent Systems.
  * Rodrigue [2013] Rodrigue Kuate, Minghua He, and Maria Chli.  An intelligent broker agent for energy trading: An mdp approach.  In Proceedings of the 2013 International Joint Conference on Artificial Intelligence, pages 234–240, 2013.
  * Tesauro [2002] Gerald Tesauro and Jonathan L. Bredin.  Strategic sequential bidding in auctions using dynamic programming.  In Proceedings of the First International Joint Conference on Autonomous Agents and Multiagent Systems: Part 2, AAMAS ’02, page 591–598, New York, NY, USA, 2002. Association for Computing Machinery.
  * Chowdhury [2018] Moinul Morshed Porag Chowdhury, Christopher Kiekintveld, Son Tran, and William Yeoh.  Bidding in periodic double auctions using heuristics and dynamic monte carlo tree search.  In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, pages 166–172. International Joint Conferences on Artificial Intelligence Organization, 7 2018\.
  * Chowdhury [2018] Moinul Morshed Porag Chowdhury, Christopher Kiekintveld, Tran Cao Son, and William Yeoh.  Bidding strategy for periodic double auctions using monte carlo tree search.  In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS ’18, page 1897–1899, Richland, SC, 2018. International Foundation for Autonomous Agents and Multiagent Systems.
  * Özdemir [2015] Serkan Özdemir and Rainer Unland.  Agentude: The success story of the power tac 2014’s champion.  In Workshop on Agent-Mediated Electronic Commerce and Trading Agent Design and Analysis (AMEC/TADA 2015), 2015.
  * Orfanoudakis [2021] Stavros Orfanoudakis, Stefanos Kontos, Charilaos Akasiadis, and Georgios Chalkiadakis.  Aiming for half gets you to the top: Winning powertac 2020.  In EUMAS, pages 144–159, 2021.
  * Tesauro [2001] Gerald Tesauro and Rajarshi Das.  High-performance bidding agents for the continuous double auction.  In Proceedings of the 3rd ACM Conference on Electronic Commerce, EC ’01, page 206–209, New York, NY, USA, 2001. Association for Computing Machinery.
  * Silver [2014] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin A. Riedmiller.  Deterministic policy gradient algorithms.  In ICML, volume 32 of JMLR Workshop and Conference Proceedings, pages 387–395. JMLR.org, 2014.
  * Puterman [1994] Martin L. Puterman.  Markov Decision Processes: Discrete Stochastic Dynamic Programming.  John Wiley and Sons, Inc., USA, 1st edition, 1994.
  * Kang [2021] Chaohai Kang, Chuiting Rong, Weijian Ren, Fengcai Huo, and Pengyun Liu.  Deep deterministic policy gradient based on double network prioritized experience replay.  In IEEE Access, 2021.