# Applications of Mechanism Design in Market-Based Demand-Side Management

Khaled Abedrabboh*  — and Luluwah Al-Fagih*  — The authors are with the
College of Science and Engineering, Hamad Bin Khalifa University, Doha 34110,
Qatar e-mail<EMAIL_ADDRESS>L. Al-Fagih is with Faculty of Science,
Engineering and Computing, Kingston University London, Kingston upon Thames
KT12EE, U.K.

###### Abstract

The intermittent nature of renewable energy resources creates extra challenges
in the operation and control of the electricity grid. Demand flexibility
markets can help in dealing with these challenges by introducing incentives
for customers to modify their demand. Market-based demand-side management
(DSM) have garnered serious attention lately due to its promising capability
of maintaining the balance between supply and demand, while also keeping
customer satisfaction at its highest levels. Many researchers have proposed
using concepts from mechanism design theory in their approaches to market-
based DSM. In this work, we provide a review of the advances in market-based
DSM using mechanism design. We provide a categorisation of the reviewed
literature and evaluate the strengths and weaknesses of each design criteria.
We also study the utility function formulations used in the reviewed
literature and provide a critique of the proposed indirect mechanisms. We show
that despite the extensiveness of the literature on this subject, there
remains concerns and challenges that should be addressed for the realistic
implementation of such DSM approaches. We draw conclusions from our review and
discuss possible future research directions.

###### Index Terms:

demand-side management; game theory; mechanism design; review; smart grid;
utility function

##  I Introduction

The global energy demand is expected to grow significantly in the near future
[1]. A move towards a larger electricity share of that demand is envisaged in
order to limit climate change and achieve sustainable development. A large
portion of this growth in demand is expected to be met by renewable energy
resources, from both large scale generation and small scale distributed
generation. However, due to the uncertainty and intermittency of renewable
energy resources, this will prompt additional challenges in maintaining the
balance between supply and demand [2]. Also, to further ensure the reliability
and security of supply, utilisation of the electricity grid infrastructure
should be enhanced. This can also help in limiting the investments required to
improve the grid infrastructure. These challenges have caused significant
interest in promising demand participation techniques.

There have been many technological developments that can allow bidirectional
power and communication flow between grid entities. Combined, these
technologies can empower the transition towards a smart grid, permitting all
grid portions to participate in the management of energy flow. The design of a
comprehensive demand side management (DSM) scheme is an essential part of the
smart grid. DSM refers to the methods used to adjust the load profile of an
electricity grid in a way that profits both supply and demand [3]. Its main
benefits include maintaining the balance between supply and demand [4] and
deferring some of the required investments in electricity infrastructure [5].
However, for these schemes to successfully engage prosumers (producer and
consumer) in abandoning some of their comfort and providing flexibility to the
grid, the design of an attractive market structure that is beneficial to all
stakeholders is paramount [6].

DSM schemes can be categorised into energy efficiency and demand response
(DR), which can refer to either price-based demand response (PBDR) or
incentive-based demand response (IBDR) ([5, 7, 8]). Fig. 1, shows the
different methods used for DSM and their classification. Energy efficiency can
either refer to improving the efficiency of demand, such as using insulation
in buildings in order to reduce the cooling/heating demand, or it can refer to
changing consumption behaviour to a more efficient one, such as refining the
thermostat temperature to reduce power consumption. This concept is often
referred to as energy conservation [9]. PBDR techniques aim to flatten the
demand profile by moving from static (demand-independent) electricity rates to
more dynamic pricing methods. These methods include Time of Use (ToU) pricing,
Critical Peak Pricing (CPP) and Real-Time Pricing (RTP). Even though these
techniques have shown positive results in demand peak shifting ([8, 10]), they
do not take the consumers’ preferences into consideration, and as a result,
customers’ responsiveness to such dynamic pricing techniques can be limited
([8, 11]). On the other hand, IBDR have provisions for customers’ individual
preferences [12]. Its programmes provide rewards for customers as an
encouragement to voluntarily participate in DR. IBDR methods can be further
categorised into classical and market-based methods. In classical IBDR,
customers are contracted to either allow control of their load to the system
operator, referred to as direct load control (DLC), or curtail their demand at
an announced DR event, referred to as interruptible/curtailable (I/C). [13].
Alternatively in market-based IBDR, customers are rewarded if they choose to
participate in a DR event [5]. Emergency DR, for instance, offers performance-
based incentives to customers who voluntarily curtail their demand in
contingencies [13]. In capacity market programmes, participants can choose to
commit to undertake a specified amount of demand reduction when called upon
[5]. Ancillary services market programmes are often offered to large consumers
or demand response aggregators (DRA) who can bid their load curtailment
services in the spot market. Accepted bids are then put on standby and
participants who are called upon need to provide their load curtailment
services at a short notice [13]. Demand bidding is arguably the most promising
method in DR as it allows customers to actively participate in the electricity
market [12]. Here, customers can individually or cooperatively bid for either
their total demand schedule or their offered demand variation to the system
operator [14]. Demand bidding can increase the price elasticity of demand and
revolutionise the electricity market, especially when distributed energy
resources such as distributed RE and electricity storage systems are used
[12].

Figure 1: Categorisation of DSM methods, highlighting the scope of this
review.

Many DSM models are based on game theoretic approaches ([6, 15]). These build
on the fact that consumers (and prosumers) act rationally and selfishly, i.e.
make logical decisions that are in their best self-interest. Indeed, game
theory can help in analysing such behaviour in competitive public resource
markets, such as transportation [16], healthcare [17] and energy ([18, 19]).
Mechanism design (MD) is the normative part of game theory, where the rules
(mechanism) of the market are not given, but designed to achieve a certain
objective [20]. Indeed, researchers have drawn parallels between the setting
of an MD problem and that of DSM. Its ability to achieve social optimal
outcomes in auctions and public good markets has given rise to the application
of MD in DSM schemes, especially demand bidding IBDR [21]. Although there are
several recent reviews available in the literature on game theoretic DSM ([22,
3, 23]), none are dedicated to MD approaches, despite the extensive research
that has been done in this field.

In this paper, we review MD models that are proposed for demand bidding IBDR.
We present different criteria for the classification of such models and
discuss the advantages/disadvantages of each class. We also review the
formulations and assumptions pertaining to the utility function used in the
proposed mechanisms. Furthermore, we discuss the potential of indirect
mechanisms in DSM schemes and provide a critique of the ones proposed in the
literature. Based on this review, we draw conclusions and discuss potential
future research directions. Our contributions are threefold:

####  I-1 classification of MD applications in DSM

We classify the available literature on DSM schemes that adopt MD in their
methods based on four criteria, revelation, allocation, sequence and
scalability. We discuss the importance of the used criteria and analyse the
properties and strengths/weaknesses of each design category. The
classification of MD-DSM schemes can provide insights into the areas that are
not thoroughly investigated yet and that would require further research.

####  I-2 survey of utility functions

We provide an in-depth examination of the utility function types used in the
existing literature that is within the scope of this review. To the best of
our knowledge, this is the first time a detailed survey of the utility
function for energy is provided. Three types of utility functions were found
in the reviewed papers. The limitations of these types are discussed in
detail.

####  I-3 review of indirect mechanisms

A promising subfield of mechanism design applications in DSM is the employment
of indirect mechanisms. A comprehensive review of DSM models that adopt such
mechanisms is provided, where we discuss the concerns that are associated with
such mechanisms.

To be able to understand the concepts and definitions used throughout this
review, the basic principles of MD are presented in Section II. In Section
III, we classify the MD mechanisms proposed for DSM and provide a critique of
each category. The types of utility functions used in these mechanisms are
then investigated in Section IV. Section V provides a thorough review of the
indirect mechanisms that are proposed for DSM. Conclusions and future research
directions are discussed in Section VI.

##  II Background on Mechanism Design

Mechanism design is the normative part of game theory. It is relevant
(applicable) when a principal is assigned with setting the rules of the market
so that a desired economic objective can be achieved [20]. One of the common
applications of MD is the pricing and provision of public goods. Given the
opportunity, consumers acting in their own self-interest would exploit the
public resource. This is known as the tragedy of the commons [24]. To avoid
this social dilemma, the mechanism designer (i.e. principal) wishes to choose
a socially preferred outcome (allocations of the public good) that guarantees
the best social economic welfare. This can be a difficult task considering
that the mechanism designer does not know the preferences of the market
participants and how much they value this public good. Another example of this
is an auction, where the principal (auctioneer) does not know the
participants’ private valuation of the auctioned items. Nonetheless, the
auctioneer’s objective is to allocate the auctioned goods to the bidders who
value them the most. Due to the preferences being privately known to the
market participants and unknown to the mechanism designer, MD has a similar
setting to that of Bayesian games, in which players have incomplete
information about other players. In order to understand the tools that are
available to the mechanism designer, we first need to analyse the setting of a
Bayesian game [25], which has:

  1. 1.

A set of agents (i.e. players) $\mathcal{N}=\\{1,\dots,n\\}$, where $n$ is the
total number of market participants. All agents are assumed to be selfish and
rational, i.e. utility maximising, where utility is the benefit they get from
consuming a product or hiring a service.

  2. 2.

A set of possible joint type vectors,
$\Theta=\Theta_{1}\times\dots\times\Theta_{n}$. An agent’s type describes
their private valuation of a good or service, however it can be extended to
include any information that is private and that can indicate their
preferences.

  3. 3.

A set of outcomes, $\Phi$. An outcome $\phi$ can include a set of allocations
$\mathcal{Q}$ and a set of monetary transfers $\mathcal{T}$. It can represent
the decisions made by the mechanism designer centrally or it can be the result
of the actions made by the agents in a distributed manner.

  4. 4.

A set of utility functions $u=\\{u_{1},\dots,u_{n}\\}$, where
$u_{i}:\Phi\times\Theta\mapsto\mathrm{I\\!R}$. A utility function is a
mathematical representation of an agent’s preferences and is a function of
consumption quantity and their type. Therefore, $u_{i}(q_{i},\theta_{i})$ is
the utility agent $i$ of type $\theta_{i}$ gets from consuming allocation
$q_{i}$ as part of an outcome $\phi$. Utility also depends on the payment
$t_{i}$ an agent makes for their consumption and their overall utility becomes
$u_{i}(q_{i},\theta_{i})-t_{i}$.

The mechanism designer’s objective is to implement a social choice function
which maps the preferences of all agents within the society to a certain
outcome. However, given that these preferences are private information, the
mechanism designer needs to elicit this information by defining the rules of
the mechanism $\mathcal{M}=\left\langle\mathcal{B},\pi,\mu\right\rangle$,
which has: (1) a message (bid) space $\mathcal{B}$, representing the set of
possible bids, (2) an allocation rule $\pi$ that maps the agents’ preferences
(bids) to an outcome $\pi:\mathcal{B}\mapsto\Phi$, and (3) a payment rule
$\mu$ that dictates, based on the agents’ bids, the amount of monetary
transfer each agent has to make $\mu:\mathcal{B}\rightarrow\mathrm{I\\!R}^{N}$
[26].

In order to understand the properties of different mechanisms, the following
definitions are presented and briefly discussed. These definitions are
reproduced from ([27, 26, 25, 28]).

Definition 1: Economic Efficiency. A mechanism is called efficient if its
allocation rule maximises the social welfare of its agents. Social welfare is
the aggregate utilities of participating agents $\sum_{i}^{N}u_{i}$, or:

$q^{\ast}(\theta)=argmax\sum_{i\in\mathcal{N}}u_{i}\left(q_{i},\theta_{i}\right)$
(1)

Definition 2: Direct Revelation. The product allocations that guarantee
maximum Social welfare can be determined by solving the optimisation problem
in Eq. 1. However, the mechanism designer does not have access to the private
preferences that are essential to solving this problem. In a direct revelation
mechanism, the principal simply asks the agents to report their private
information as part of their bid space. This simplifies the mechanism design
problem, however, given the selfish nature of individual agents, it raises
another concern that agents might be tempted to report their private types
untruthfully. As false reporting would compromise the efficiency of the
mechanism, it is imperative that agents are incentivised to report their types
truthfully. This is known as the incentive compatibility constraint (see
definition below). In contrast, indirect mechanisms are privacy-preserving
mechanisms that do not require agents to fully report their private
information. This can be done by collecting incremental information from the
agents in a multiround (i.e. iterative) mechanism.

Definition 3: Incentive Compatibility. This constraint is essential to ensure
truthfulness in direct mechanisms and guarantee economic efficiency. To ensure
that agents, acting in their own self-interest, report their private types in
a truthful manner, the mechanism designer must assert that agents do not gain
from false bidding. The incentive compatibility constraint is formulated as:

$u_{i}(q_{i},\theta_{i})\geq u_{i}(q_{i},\theta)~{}~{}~{}\forall
i\in\mathcal{N}$ (2)

This means that the utility $u_{i}(q_{i},\theta_{i})$ agent $i$ gets from
reporting their true type $\theta_{i}$ is never less than what they get from
untruthful bidding $u_{i}(q_{i},\theta)$.

Definition 4: Individual Rationality. Individual rationality (IR), or
participation constraint is a desired property of mechanisms in which
participation is optional. It ensures that agents receive at least as much
utility as they would get by not taking part. Assuming that an agent gets a
payoff of zero by non-participation, IR constraint can be formulated as [28]:

$u_{i}(q_{i},\theta_{i})\geq 0~{}~{}~{}\forall i\in\mathcal{N}$ (3)

Definition 5: Budget Balance. To prevent monetary losses from being incurred
by the mechanism, the coordinating agent (i.e. mechanism designer) should
select a mechanism that guarantees budget balance, or
$\sum_{i}^{N}t_{i}(\theta)\geq 0$. This means that the mechanism does not
require external subsidies as it either generates money or preserve its
balance.

##  III classification

Mechanism design can help in selecting social outcomes that avoid
inefficiencies in the allocation of a good or service. For this reason, many
scientists have adopted MD techniques in their approaches to DSM. To the best
of our knowledge, [29] is the first work that proposes an MD approach to DSM.
Since then, the research on DSM using MD has been extensive. In order to have
an overview of the different mechanisms proposed for DSM, we investigate four
criteria, based on which we categorise all the proposed models. These four
classes are discussed below. Table I lists the proposed mechanisms in their
respected categories.

TABLE I: Classification of reviewed mechanisms based on criteria described in section III. |  | revelation | allocation | sequence | scalability
---|---|---|---|---|---
# papers | ref |  direct |  indirect |  central |  distributed |  iterative |  one-shot |  scalable |  unscalable
27 | [29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56] | ✓ |  | ✓ |  |  | ✓ |  | ✓
21 | [57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77] | ✓ |  | ✓ |  |  | ✓ | ✓ |
3 | [78, 79, 80] | ✓ |  | ✓ |  | ✓ |  | ✓ |
3 | [81, 82, 83] | ✓ |  |  | ✓ |  | ✓ |  | ✓
13 | [84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96] | ✓ |  |  | ✓ |  | ✓ | ✓ |
3 | [97, 98, 99] | ✓ |  |  | ✓ | ✓ |  | ✓ |
1 | [100] |  | ✓ | ✓ |  | ✓ |  |  | ✓
3 | [101, 102, 103] |  | ✓ | ✓ |  | ✓ |  | ✓ |
13 | [104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116] |  | ✓ |  | ✓ | ✓ |  | ✓ |
5 | [117, 118, 119, 120, 121] |  | ✓ |  | ✓ |  | ✓ | ✓ |

####  III-1 Revelation

One of the key factors in mechanism design is whether agents are required to
report their private preferences directly and fully. These direct mechanisms
are used to simplify the principal’s task of implementing a social choice
function. They can also guarantee the same level of efficiency (competitive
ratio) as any other indirect mechanism. In fact, the revelation principle
([122, 123]) states that if a social choice function can be implemented by any
mechanism, it can be implemented by a direct one without any loss of payoff.
Nonetheless, direct mechanisms become less desirable in settings where agents
prefer to keep their information private. This is a limitation when designing
mechanisms for DSM as consumers may be reluctant to share their demand
information and their value of that demand. Another limitation of direct
mechanisms is that the communication overhead can become inefficient in
markets that have a large number of agents, especially when multi-dimensional
preferences are elicited. This concern can hinder the feasibility of direct
mechanisms for DSM as most DSM applications need to engage a large number of
consumers to ensure a certain level of demand flexibility. This communication
overhead becomes aggravated as many DSM mechanisms ask agents to report their
day-ahead consumption schedules and sometimes even ask for appliance-based
demand information. Furthermore, computational complexity can be a concern in
direct mechanisms because all the computation burden of solving the social
welfare optimisation problem and determining the allocations/payments for each
agent is incurred by the mechanism designer. This can limit the scalability of
such mechanisms.

Indirect mechanisms on the other hand are privacy preserving in the sense that
agents are not required to share their information fully. Indeed, a social
choice function can be implemented through an indirect mechanism by collecting
sequential bids from agents without revealing their complete preferences.
Although indirect mechanisms tend to be tractable as they can be implemented
in a distributed fashion, they generally suffer from efficiency losses. A
thorough review of the indirect mechanisms for DSM is provided in Section V.

####  III-2 Allocation

Another key factor in the design of DSM mechanisms is the architecture used in
their supply chain system. A central architecture refers to a supply chain
system where a central entity (the principal) makes allocation decisions based
on its interactions with its agents. This can be inconvenient for consumers
who are not equipped with distributed resources or storage systems, and
although most of the mechanisms that adopt this architecture can optimise
social welfare and yield efficient outcomes, adopting them in practice might
result in significant discomfort for the consumers. Additionally, electricity
consumers are usually unable to forecast their consumption accurately. Because
of this, central allocation mechanisms are often coupled with reward/penalty
incentives, which depend on how accurately consumers follow their assigned
allocations. This uncertainty in demand does not only harm the efficiency of
such mechanisms but also limit their acceptance among consumers.
Alternatively, distributed mechanisms adopt an architecture where agents make
their allocation decisions locally. In this architecture both the mechanism
designer and the agents solve for the allocations and payments that optimise
social welfare, thus sharing the computational burden of the mechanism.
Although this architecture can ease the computation of DSM mechanisms and is
more favorable to consumers than the central allocation architecture, it
raises concerns about how faithfully agents choose and implement their
allocation schedules.

####  III-3 Sequence

Some of the reviewed DSM mechanisms require agents to report their private
preferences in one-shot. Others, in contrast use an iterative structure where
agents update their bids at each step in response to a signal received from
the mechanism designer. This is generally implemented to either ease the
computational burden on the principal side ([97, 98, 99]) or to preserve the
privacy of agents through indirect mechanisms ([105, 102, 103, 107, 115]).
Mechanisms that adopt the one-shot bidding process are more robust to
communication delays. They can also be faster to implement. However, they are
often less practical than iterative mechanisms as it is difficult for
consumers to integrate their preferences into one bid, especially in a day-
ahead setting as proposed in most DSM schemes. Nonetheless, although iterative
mechanisms can overcome this limitation by asking agents to respond to their
signals at each iteration, they usually have two main disadvantages;
convergence in such schemes is generally slow, especially when a large number
of participants need to transmit their bids at each iteration. This can be
exacerbated when communication failures or delays are considered. The second
limitation is that experienced agents that have an estimate of the number of
iterations required for convergence can be untruthful in their bids at
intermediary iterations if it generates higher utility gains. This can
compromise the strategy proofness of such mechanisms.

####  III-4 Scalability

To maximise the benefits of DSM, broad participation of all types of demand is
encouraged. Indeed, both large and small customers should engage in demand
flexibility markets to ensure the success of such schemes [14]. Given that the
number of electricity consumers is significantly large, scalability is one of
the most important features of demand side participation mechanisms. Some
works, (e.g. [107, 98, 91, 103]), group a number of small residential
customers under one agent, who is often referred to as a demand response
aggregator (DRA). Thus proposing a two-level hierarchical structure for DSM
where small customers and the DRA interact at the lower level, while DRAs
offer their flexibility services to the grid at the higher level. Although
this can significantly reduce the number of agents at the both grid and DRA
levels, the number of agents at the lower level can still be quite large
($\sim 1000$ [107]). Therefore, scalability is an essential requirement for
the practical implementation of DSM mechanisms.

One of the most well known mechanisms that can ensure efficient allocation of
goods/services is the Vickrey-Clarke-Groves (VCG) mechanism. VCG is a direct
and central mechanism, where agents report their private utilities to the
principal who then determines their allocations by solving the social welfare
optimisation problem. The payment rule in VCG is based on Clarke pivot rule,
which requires that each agent pays their social cost of participation, which
is the loss in welfare of all the other agents due to the participation of
that agent. This payment rule ensures that the best strategy for agents who
act selfishly is to report their private information truthfully. Many of the
MD-DSM approaches are based on VCG ([29, 32, 37, 61, 62, 42, 43, 68, 69, 48,
80, 50, 52, 70, 53, 73, 77, 81]). Although VCG can guarantee economic
efficiency and incentive compatibility, it raises privacy and comfort concerns
when applied to electricity market settings due to its direct revelation and
central allocation structure. Additionally, it requires that agents’ valuation
functions (i.e. utility functions) follow a strictly concave form. This does
not only limit the representation of agents’ preferences but also renders the
social welfare optimisation problem intractable [30]. VCG is also vulnerable
to collusion and shell bidding [124]. Payments can also vary widely between
agents and those with high valuations get higher allocations, which can be
considered unfair, especially in settings where consumption is essential to
preserve the quality of life such as electricity.

##  IV the utility function

Utility is a numerical measure of the satisfaction an individual gets from
consuming a good or receiving a service. The utility function is a
mathematical representation of this satisfaction as a function of the
individual’s preferences over a set of outcomes [125]. In settings where
transferable utility is possible through monetary payment, the utility
function is said to be quasi-linear, i.e. linear in payment, or reward. This
linearity in monetary transfer is depicted in Eq. 4, where $U$ refers to the
benefit a customer gets, which is represented by the difference between the
customer’s satisfaction and their payment. Alternatively, in the case that the
agent is the seller or service provider, their utility can be formulated as
the difference between the reward they get from providing that service and the
associated cost of providing it.

$U=u(q,\theta)-p$ (4)

Representing the level of satisfaction of an agent as a function of
consumption quantity and their preferences can be a difficult task. This is
because preferences tend to be multi-dimensional. For instance, the utility of
electricity consumption that is related to entertainment appliances (such as
TV) is usually dependent on the time of consumption rather than quantity of
consumption. Therefore, capturing these individual private preferences into
one mathematical formulation that is suitable to all participating agents can
be tricky. Nonetheless, the design of most mechanisms depend on making some
underlying assumptions about the utility function. Additionally, to simplify
the computation of their outcome and reduce their communication overhead, some
direct mechanisms rely on a closed formulation of the utility function that
can reduce the dimensionality of agents’ preferences. The utility function
assumptions and formulations used in the reviewed literature can be classified
into three types. Table II summarises the formulations of these three types
and lists the models that employ them. The underlying assumptions associated
with the three types used in the literature are discussed below.

TABLE II: utility function types and assumptions used in the reviewed literature |  Assumptions |  Examples |  References
---|---|---|---
Type I |  twice differentiable, monotonically increasing and strictly concave |  quadratic, logarithmic |  [31, 118, 33, 35, 106, 40, 41, 45, 101, 109, 110, 111, 51, 93, 29, 32, 81, 34, 44, 67, 91, 83, 92, 99, 80, 108, 119, 78, 112, 30, 46, 103, 113, 114, 115, 116, 76]
Type II |  piecewise linear and concave |  - |  [117, 120, 94, 96]
Type III |  constant utility if preferences are satisfied and zero otherwise |  - |  [87, 98]

###  IV-A Type I: strictly concave, twice differentiable utility function

The most commonly used formulation for the utility function is that it is
strictly concave in consumption quantity. This formulation is based on the
assumption that the utility function should satisfy the following properties:

  1. 1.

The utility function $U(q,\theta)$ is monotonically increasing in $q$, i.e.
more consumption yields higher utility.

$\frac{\partial U(q,\theta)}{\partial q}\geq 0$ (5)

  2. 2.

The marginal utility is nonincreasing in consumption quantity. This stems from
the assumption that the larger the stock a person has the less value they
would get from a given increase to that stock, i.e. as their stock increases,
their utility increases at a diminishing rate. The law of diminishing marginal
utility is discussed at length in [126].

$\frac{\partial^{2}U(q,\theta)}{\partial q^{2}}\leq 0$ (6)

Strictly concave functions satisfy the above properties. Examples of such
functions include a bounded quadratic function, used in ([29, 32, 81, 34, 44,
67, 91, 92, 83, 99, 80, 108, 103, 114, 116]) and a logarithmic function ([119,
78, 112, 113]). Many of the reviewed papers model the utility function to be
concave in quantity, however, this formulation has some limitations. One of
the conditions that needs to hold for this formulation to be valid is that an
agent’s tastes and preferences do not change during the time-cycle of the
mechanism. This can be challenging in the electricity market setting because
most of the reviewed models propose day-ahead bidding and scheduling of
electricity consumption. Additionally, the law of diminishing utility may not
hold when dealing with task-related consumption, such as the case of
electricity consumption. This is visible when an example of task related
electricity consumption such as cooking is considered. Imagine that a meal
would require 10 kWh to be done. consuming the first 9 kWh would have no value
unless the last kWh is consumed. Therefore the utility rate of rise in such a
scenario would be increasing with an increase in consumption rather than
decreasing. This limitation is discussed at length in [126]. Additionally, one
of the underlying assumptions for this formulation to hold is that electricity
consumption is continuous. In reality, electricity consumption is task-related
and is therefore a mixture of discrete and continuous energy demand as argued
in [59]. Furthermore, this type of valuation function does not take into
account the intertemporal dependence of electricity consumption. This can lead
to misrepresenting the preferences of an agent since they might change
depending on past or future consumption [87].

###  IV-B Type II: piecewise linear concave utility function

The piecewise linear utility function is an extension of the concave utility
function. The difference is that this type of utility functions is non-
differentiable whereas strictly concave functions are twice differentiable.
Indeed, piecewise linear utility functions are used in ([117, 120]) as a
special case of concave utility functions so that the incentive compatibility
property can hold in their proposed mechanism. The authors show that agents
with non-constant marginal utility might gain (benefit) from being untruthful
in reporting their private preferences. The authors in [94] also use a
piecewise linear utility function to model the agent’s satisfaction from
electricity consumption. In their proposed mechanism, an agent is incentivised
to report their baseline consumption and their constant marginal utility
truthfully, upon which, the demand response aggregator (DRA) selects the
agents that can achieve a demand reduction target in a cost efficient way.

###  IV-C Type III: constant utility function

The authors in [87, 98] argue that continuous concave utility functions cannot
represent the task-related and combinatorial nature of electricity
consumption. Instead, they use a discrete utility function that is constant if
the reported consumer’s preferences are satisfied and zero otherwise. In their
proposed mechanism, household agents report their appliances’ energy
requirements and time flexibility. These preferences are then used to minimise
the overall cost of energy. Although the intertemporal dependence of
electricity consumption is captured in this mechanism, the dissatisfaction
from shifting the operation of an appliance across the reported flexibility is
not.

Despite types II and III being simple and computationally efficient, they fail
to capture the complexity of consumption preferences. Consumers, therefore,
might choose not to participate in the DSM schemes that force such utility
formulations as they fail to represent their valuation of electricity
consumption.

##  V Review of indirect mechanisms

Indirect mechanisms are capable of achieving desired social objectives without
the need for a full revelation of the agents’ private preferences. They become
extremely important when these preferences are multi-dimensional and when
market players are hesitant to participate because they are reluctant to
reveal their private information. In these mechanisms, agents are required to
provide incremental information that can indicate their private preferences.
As a result, most indirect mechanisms are iterative in the sense that agents
are required to update their bids at each step of the mechanism. An outcome is
reached when a convergence criterion is met. Table I lists the indirect
mechanisms that were proposed for DSM in the literature. Some of the concerns
that accompany indirect mechanisms are discussed below.

####  V-1 Efficiency

Since solving the optimal social welfare problem (stated in Eq. 1) requires
the full knowledge of the private utilities of the participating agents,
indirect mechanisms usually suffer from efficiency losses because they only
ask for partial information about these private utilities. In ([117, 120]) for
instance, the authors propose a DSM mechanism where a supplier offers a non
increasing deadline differentiated pricing bundle (i.e. lower price for demand
with further deadline), to which customers respond by reporting their deadline
differentiated demand. The optimal pricing bundle is then determined based on
the supplier’s optimal supply curve and the expected RE generation. The
mechanism, however, makes an underlying assumption that customers are
indifferent whether their demand is met before or at their reported deadline.
This is likely not the case for most customers who would appreciate completing
their tasks at the earliest possible. Because of this, losses in the welfare
of users might arise, rendering the mechanism inefficient and unappealing in
practice.

The authors in ([101, 102]) propose a double auction for energy trading
between prosumers in a microgrid. The coordinator in this mechanism announces
initial demand allocations to the consumers and initial supply requirements
and price to the producers. Buyers then bid their optimal prices for the
allocated demand and suppliers bid their optimal energy supply at the
announced price. The coordinator then updates the market price and energy
allocations by solving SWO. This is repeated until a convergence criterion is
met. One of the limitations of this mechanism is that agents are assumed to be
price taking, whose actions do not influence market price. In reality however,
agents can be strategic and might know their actions’ effect on the market.
Another limitation of this mechanism is that final allocations are determined
centrally. This can lead to efficiency losses if agents choose not to follow
these allocations due to uncertainty in supply or demand. A posted price
mechanism for energy markets is presented in [121]. The authors design optimal
price profiles for each time interval that agents (arriving at different
stages) can take or leave. Although the mechanism proves to be robust against
uncertainties in demand and is privacy preserving, it suffers from losses in
efficiency due to the partial knowledge of agents’ preferences. The authors in
[113] propose a pricing mechanism for EV charging stations by employing a
Markov decision process to model the strategic interactions between the
station and its connected EVs. Although this pricing mechanism can maximise
the long-term revenue of the charging station, it does not however maximise
the social welfare of its users.

####  V-2 Convergence

Given that most of the indirect mechanisms proposed for DSM are iterative,
evaluating the convergence speed of such mechanisms is essential to their
successful implementation in practice. The authors in [100] propose a double
auction mechanism for energy transactions between prosumers in a micro grid,
where buyers and sellers bid their prices for each opposing agent. The micro
grid controller then determine the demand and supply vectors that optimise the
social welfare, based on which, agents update their bids in an iterative and
distributed manner. Given that agents are required to solve a number of
optimisation problems at every step of the mechanism, convergence to the
optimal allocations might be slow, which would limit the proposed auction from
being implemented in practice.

An RTP scheme for prosumers in a microgrid is proposed in [106, 115], where
the authors use subsidies to align selfish behaviour with social behaviour.
These subsidies are computed through an iterative power and price information
exchange between the agents. A limitation of this mechanism is that
convergence cannot be guaranteed unless the private utilities of the agents
are known. The authors in [110] propose a mechanism for strategic agents that
can provide flexibility services to the grid. The authors use a reward/penalty
billing rule that penalises demand that is highly correlated with aggregate
demand rewards it when it is less correlated. In this mechanism, an iterative
simulated annealing method is used to determine the agents’ payments, where
agents need to to solve a set of optimisation problems at each iteration. This
method can be unpredictable and slow in terms of rate of convergence. In
[111], the authors propose an RTP pricing scheme that is customised for each
user, which offers lower prices for users who consume a less percentage of
their desired level and higher prices for those who consume a higher
percentage of their desired consumption level. Although social welfare can be
maximised using this scheme, the prices are determined in an iterative manner
where each agent reports their optimal consumption in response to the updated
price at each step. This procedure needs to be performed for each agent, which
can lead to high computational complexity and slow convergence.

A bilevel auction mechanism for demand response aggregators (DRA) is proposed
in [103]. In this mechanism, DRAs bid for their energy share at the higher
level, and prosumers bid for their supply/demand at the lower level. At each
step of this auction, aggregators receive their allocated energy information
from the utility company and implement their lower level auction by
interacting with their selling and buying agents sequentially and separately.
The complexity of this 2-level auction format along with the asymmetry in
bidding between sellers and buyers in the lower level auction might cause
convergence to be slow.

####  V-3 Privacy

Indirect mechanisms can overcome privacy concerns of consumers who do not wish
to reveal their private valuation information fully by asking for partial
incremental information about their private preferences. Nonetheless, in some
of the indirect mechanisms that are iterative, the agents’ updated reports
might be exploited to reveal some of their private valuations and preferences.
Moreover, some of the indirect mechanisms request their agents to report their
desired consumption schedule given an announced price. This can be more
unacceptable to some consumers than revealing their private valuation. A
clock-proxy auction mechanism for online scheduling of demand is proposed in
[107]. In this 2-phase auction format, (ball park prices are agreed upon in
the clock phase) price discovery is implemented in the clock phase with
iterative price adjustments and bid updates. Whereas the final uniform time-
dependent prices and demand schedules are determined for each time interval in
the proxy phase. Although this auction does not require agents to reveal their
private information fully, a closer look into their bid updates can indicate
an approximate representation of their utilities, thus compromising their
privacy. In ([109, 112]), the authors propose incentive mechanisms that
encourage social consumption behaviour. In both mechanisms, agents update
their consumption schedules sequentially in response to an announced price or
power signal. This might be limiting in practice as consumers would be
reluctant to share their private consumption patterns.

####  V-4 Intermediate False Reporting

One of the major concerns that need to be investigated before iterative
mechanisms can be applied in practice is unfaithful reporting in intermediary
steps. This refers to when experienced agents who have an estimate of the
number of iterations required for convergence may submit false bids in the
iterations before the final one if it is in their self-interest. This concern
becomes evident when we consider the mechanisms proposed in ([104, 105, 108,
116]), where an iterative exchange of price and power information is used to
determine the optimal price and demand. The authors, however, do not
investigate whether agents reporting their demand falsely in intermediary
iterations can achieve higher utility gains than truthfully reporting their
demand at every iteration.

####  V-5 Consumption-Time Dependence

Some of the proposed indirect mechanisms for DSM fail to model the temporal
dependence of electricity consumption. In ([118, 119]) for instance, an
incentive mechanism is proposed to modify the consumption behaviour of selfish
customers. In this mechanism, customers’ consumption schedules are used to
compute their customised incentives. The authors, however, assume that the
utility customers get from consuming energy at a given time does not depend on
their consumption prior to that time or their planned consumption after it.
This can be limiting since customers usually value their consumption
differently depending on their previous and future planned consumption. The
authors in [114] propose a DSM mechanism where customers in a neighborhood
cooperatively estimate the real-time price of the future time slot and then
schedule their consumption accordingly to achieve their best self-interest.
The regressors used in the estimation method, which are the historical demand
profiles of the neighborhood agents, are assumed to be temporally independent.
This underlying assumption may lead to inefficiencies due to the temporal
dependence of electricity consumption.

##  VI Conclusion

In this paper, MD applications in market-based DSM were reviewed. the
available literature were classified according to four design criteria; (1)
direct or indirect revelation, (2) central or distributed allocation decision
making, (3) iterative or one-shot bidding sequence, and (4) scalability of the
proposed mechanisms. These design tools were investigated and their
benifits/drawbacks in DSM applications were analysed. A challenging
preliminary in many of the proposed MD-DSM approaches is the mathematical
formulation of the utility function. Three types of this formulation were
found in the literature within the scope of this review. The limitations and
concerns of each type of formulation were discussed. As a promising subfield
of MD applications in DSM, indirect revelation mechanisms were thoroughly
reviewed and the concerns associated with these mechanisms were investigated.
One of the areas that requires further research is the design of indirect,
non-iterative mechanisms for DSM that adopt a distributed allocation
architecture and that is computationally simple. Another future research
direction is the design of combinatorial auctions, which combine a number of
goods and services into one item, for DSM. This can be promising because
prosumers may value a combination of generation, consumption, storage and
demand flexibility services differently when compared to each good or service
separately. Due to privacy concerns, further research into DSM mechanisms that
can offer optional revelation schemes for electricity consumers/prosumers may
be required to attract wider customers’ participation.

## References

  * [1] International Energy Agency (IEA), “World energy outlook 2020.” [Online]. Available: https://www.iea.org/reports/world-energy-outlook-2020
  * [2] S. Impram, S. Varbak Nese, and B. Oral, “Challenges of renewable energy penetration on power system flexibility: A survey,” _Energy Strategy Reviews_ , vol. 31, p. 100539, 2020.
  * [3] R. Sharifi, S. Fathi, and V. Vahidinasab, “A review on demand-side tools in electricity market,” _Renewable and Sustainable Energy Reviews_ , vol. 72, pp. 565–572, 2017.
  * [4] V. S. K. Murthy Balijepalli, V. Pradhan, S. A. Khaparde, and R. M. Shereef, “Review of demand response under smart grid paradigm,” in _ISGT2011-India_ , 2011, pp. 236–243.
  * [5] M. Albadi and E. El-Saadany, “A summary of demand response in electricity markets,” _Electric Power Systems Research_ , vol. 78, no. 11, pp. 1989–1996, 2008.
  * [6] V. Chrysikou, M. Alamaniotis, and L. H. Tsoukalas, “A review of incentive based demand response methods in smart electricity grids,” _International Journal of Monitoring and Surveillance Technologies Research (IJMSTR)_ , vol. 3, no. 4, pp. 62–73, 2015.
  * [7] P. Palensky and D. Dietrich, “Demand side management: Demand response, intelligent energy systems, and smart loads,” _IEEE Trans. Industrial Informatics_ , vol. 7, no. 3, pp. 381–388, 2011.
  * [8] X. Yan, Y. Ozturk, Z. Hu, and Y. Song, “A review on price-driven residential demand response,” _Renewable and Sustainable Energy Reviews_ , vol. 96, pp. 411–419, 2018.
  * [9] F. Boshell and O. P. Veloza, “Review of developed demand side management programs including different concepts and their results,” in _2008 IEEE/PES Transmission and Distribution Conference and Exposition: Latin America_ , 2008, pp. 1–7.
  * [10] D. Srinivasan, S. Rajgarhia, B. M. Radhakrishnan, A. Sharma, and H. Khincha, “Game-theory based dynamic pricing strategies for demand side management in smart grids,” _Energy_ , vol. 126, pp. 132–143, 2017.
  * [11] P. C. Reiss and M. W. White, “Household Electricity Demand, Revisited,” _The Review of Economic Studies_ , vol. 72, no. 3, pp. 853–883, 07 2005.
  * [12] Q. Zhang and J. Li, “Demand response in electricity markets: A review,” in _2012 9th International Conference on the European Energy Market_ , 2012, pp. 1–8.
  * [13] P. Khajavi, H. Abniki, and A. B. Arani, “The role of incentive based demand response programs in smart grid,” in _2011 10th International Conference on Environment and Electrical Engineering_ , 2011, pp. 1–4.
  * [14] N. G. Paterakis, O. Erdinç, and J. P. Catalão, “An overview of demand response: Key-elements and international experience,” _Renewable and Sustainable Energy Reviews_ , vol. 69, pp. 871–891, 2017.
  * [15] B. P. Esther and K. S. Kumar, “A survey on residential demand side management architecture, approaches, optimization models and methods,” _Renewable and Sustainable Energy Reviews_ , vol. 59, pp. 342–351, 2016.
  * [16] H. M. Amar and O. A. Basir, “A game theoretic solution for the territory sharing problem in social taxi networks,” _IEEE Trans. Intelligent Transportation Systems_ , vol. 19, no. 7, pp. 2114–2124, 2018.
  * [17] K. Abedrabboh, M. Pilz, Z. Al-Fagih, O. S. Al-Fagih, J.-C. Nebel, and L. Al-Fagih, “Game theory to enhance stock management of personal protective equipment (ppe) during the covid-19 outbreak,” _PLOS ONE_ , vol. 16, no. 2, pp. 1–21, 02 2021.
  * [18] M. Pilz, L. Al-Fagih, and E. Pfluegel, “Energy storage scheduling with an advanced battery model: a game–theoretic approach,” _Inventions_ , vol. 2, no. 4, p. 30, 2017.
  * [19] M. Pilz and L. Al-Fagih, “A dynamic game approach for demand-side management: scheduling energy storage with forecasting errors,” _Dynamic Games and Applications_ , pp. 1–33, 2019.
  * [20] D. Fudenberg and J. Tirole, _Game Theory_. Cambridge, MA: MIT Press, 1991.
  * [21] S. Li, J. Lian, A. J. Conejo, and W. Zhang, “Transactive energy systems: The market-based coordination of distributed energy resources,” _IEEE Control Systems Magazine_ , vol. 40, no. 4, pp. 26–52, 2020.
  * [22] M. Pilz and L. Al-Fagih, “Recent advances in local energy trading in the smart grid based on game-theoretic approaches,” _IEEE Trans. Smart Grid_ , vol. 10, no. 2, pp. 1363–1371, 2019.
  * [23] A. Loni and F. Parand, “A survey of game theory approach in smart grid with emphasis on cooperative games,” in _2017 IEEE International Conference on Smart Grid and Smart Cities (ICSGSC)_ , 2017, pp. 237–242.
  * [24] G. Hardin, “The tragedy of the commons,” _Journal of Natural Resources Policy Research_ , vol. 1, no. 3, pp. 243–253, 2009.
  * [25] M. O. Jackson, “Mechanism theory,” _Available at SSRN_ , 2014.
  * [26] V. Krishna, “Chapter five - mechanism design,” in _Auction Theory (Second Edition)_ , second edition ed., V. Krishna, Ed. San Diego: Academic Press, 2010, pp. 61–83.
  * [27] N. Nisan, T. Roughgarden, E. Tardos, and V. Vazirani, _Algorithmic Game Theory_. Cambridge University Press, 2007\.
  * [28] Y. Narahari, _Game Theory and Mechanism Design_. WORLD SCIENTIFIC / INDIAN INST OF SCIENCE, INDIA, 2014.
  * [29] P. Samadi, R. Schober, and V. W. S. Wong, “Optimal energy consumption scheduling using mechanism design for the future smart grid,” in _2011 IEEE International Conference on Smart Grid Communications (SmartGridComm)_ , 2011, pp. 369–374.
  * [30] D. Li, S. K. Jayaweera, and A. Naseri, “Auctioning game based demand response scheduling in smart grid,” in _2011 IEEE Online Conference on Green Communications_ , 2011, pp. 58–63.
  * [31] J. Cao, B. Yang, C. Chen, and X. Guan, “Optimal demand response using mechanism design in the smart grid,” in _Proceedings of the 31st Chinese Control Conference_ , 2012, pp. 2520–2525.
  * [32] P. Samadi, H. Mohsenian-Rad, R. Schober, and V. W. S. Wong, “Advanced demand side management for the future smart grid using mechanism design,” _IEEE Trans. Smart Grid_ , vol. 3, no. 3, pp. 1170–1180, 2012.
  * [33] Y. Chen, W. S. Lin, F. Han, Y.-H. Yang, Z. Safar, and K. J. R. Liu, “Incentive compatible demand response games for distributed load prediction in smart grids,” _APSIPA Trans. Signal and Information Processing_ , vol. 3, p. e9, 2014.
  * [34] J. Ma, J. Deng, L. Song, and Z. Han, “Incentive mechanism for demand side management in smart grid using auction,” _IEEE Trans. Smart Grid_ , vol. 5, no. 3, pp. 1379–1388, 2014.
  * [35] E. Salfati and R. Rabinovici, “Demand-side management in smart grid using game theory,” in _2014 IEEE 28th Convention of Electrical Electronics Engineers in Israel (IEEEI)_ , 2014, pp. 1–5.
  * [36] S. Mhanna, G. Verbič, and A. C. Chapman, “Towards a realistic implementation of mechanism design in demand response aggregation,” in _2014 Power Systems Computation Conference_ , 2014, pp. 1–7.
  * [37] N. Z. Xu and C. Y. Chung, “Challenges in future competition of electric vehicle charging management and solutions,” _IEEE Trans. Smart Grid_ , vol. 6, no. 3, pp. 1323–1331, 2015.
  * [38] D. Li and S. K. Jayaweera, “Distributed smart-home decision-making in a hierarchical interactive smart grid architecture,” _IEEE Trans. Parallel and Distributed Systems_ , vol. 26, no. 1, pp. 75–84, 2015.
  * [39] K. Hayakawa, E. Gerding, S. Stein, and T. Shiga, “Online mechanisms for charging electric vehicles in settings with varying marginal electricity costs,” in _IJCAI’15 Proceedings of the 24th International Conference on Artificial Intelligence_. ACM, July 2015, pp. 2610–2616.
  * [40] S. Li, W. Zhang, J. Lian, and K. Kalsi, “Market-based coordination of thermostatically controlled loads—part i: A mechanism design formulation,” _IEEE Trans. Power Systems_ , vol. 31, no. 2, pp. 1170–1178, 2016.
  * [41] ——, “Market-based coordination of thermostatically controlled loads—part ii: Unknown parameters and case studies,” _IEEE Trans. Power Systems_ , vol. 31, no. 2, pp. 1179–1187, 2016.
  * [42] S. Bistarelli, R. Culmone, P. Giuliodori, and S. Mugnoz, “Mechanism design approach for energy efficiency,” 2016.
  * [43] P. Giuliodori, “A mechanism design approach for energy allocation,” in _DC@AI*IA_ , 2016.
  * [44] A. Sinha and A. Anastasopoulos, “Incentive mechanisms for fairness among strategic agents,” _IEEE Journal on Selected Areas in Communications_ , vol. 35, no. 2, pp. 288–301, 2017.
  * [45] H. Tavafoghi and D. Teneketzis, “Multidimensional forward contracts under uncertainty for electricity markets,” _IEEE Trans. Control of Network Systems_ , vol. 4, no. 3, pp. 511–522, 2017.
  * [46] H. Ma, D. C. Parkes, and V. Robu, “Generalizing demand response through reward bidding,” in _Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems_ , 2017, pp. 60–68.
  * [47] Y. Li and N. Li, “Mechanism design for reliability in demand response with uncertainty,” in _2017 American Control Conference (ACC)_ , 2017, pp. 3400–3405.
  * [48] T. Tanaka, N. Li, and K. Uchida, “On the relationship between the vcg mechanism and market clearing,” in _2018 Annual American Control Conference (ACC)_ , 2018, pp. 4597–4603.
  * [49] D. Li, Q. Yang, W. Yu, D. An, X. Yang, and W. Zhao, “A strategy-proof privacy-preserving double auction mechanism for electrical vehicles demand response in microgrids,” in _2017 IEEE 36th International Performance Computing and Communications Conference (IPCCC)_ , 2017, pp. 1–8.
  * [50] W. Zhong, K. Xie, Y. Liu, C. Yang, and S. Xie, “Auction mechanisms for energy trading in multi-energy systems,” _IEEE Trans. Industrial Informatics_ , vol. 14, no. 4, pp. 1511–1521, 2018.
  * [51] S. Chen and R. S. Cheng, “Operating reserves provision from residential users through load aggregators in smart grid: A game theoretic approach,” _IEEE Trans. Smart Grid_ , vol. 10, no. 2, pp. 1588–1598, 2019.
  * [52] Y. G. F. Borges, R. C. S. Schouery, F. K. Miyazawa, F. Granelli, N. L. S. da Fonseca, and L. P. Melo, “Smart energy pricing for demand-side management in renewable energy smart grids,” _International Trans. Operational Research_ , vol. 27, no. 6, pp. 2760–2784, 2020.
  * [53] C. P. Mediwaththe, M. Shaw, S. Halgamuge, D. B. Smith, and P. Scott, “An incentive-compatible energy trading framework for neighborhood area networks with shared energy storage,” _IEEE Trans. Sustainable Energy_ , vol. 11, no. 1, pp. 467–476, 2020.
  * [54] B. Dakhil and A. Gupta, “Auctioning electricity under deep renewable integration using a penalty for shortfall,” _Sustainable Energy, Grids and Networks_ , vol. 20, p. 100266, 2019.
  * [55] D. Li, Q. Yang, W. Yu, D. An, Y. Zhang, and W. Zhao, “Towards differential privacy-based online double auction for smart grid,” _IEEE Trans. Information Forensics and Security_ , vol. 15, pp. 971–986, 2020.
  * [56] G. Tsaousoglou, J. S. Giraldo, P. Pinson, and N. G. Paterakis, “Mechanism design for fair and efficient dso flexibility markets,” _IEEE Trans. Smart Grid_ , pp. 1–1, 2020.
  * [57] R. Kota, G. Chalkiadakis, V. Robu, A. Rogers, and N. R. Jennings, “Cooperatives for demand side management,” in _The Seventh Conference on Prestigious Applications of Intelligent Systems (PAIS @ ECAI) (30/08/12)_ , August 2012, pp. 969–974.
  * [58] P. Ströhle, E. Gerding, M. de Weerdt, S. Stein, and V. Robu, “Online mechanism design for scheduling non-preemptive jobs under uncertain supply and demand,” in _AAMAS ’14 Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems_ , A. Lomuscio, P. Scerri, A. Bazzan, and M. Huhns, Eds. ACM, May 2014, pp. 437–444.
  * [59] S. Mhanna, G. Verbič, and A. C. Chapman, “Guidelines for realistic grounding of mechanism design in demand response,” in _2014 Australasian Universities Power Engineering Conference (AUPEC)_ , 2014, pp. 1–6.
  * [60] C.-K. Chau, K. Elbassioni, and M. Khonji, “Truthful mechanisms for combinatorial ac electric power allocation,” 2014.
  * [61] Q. Xiang, F. Kong, X. Liu, X. Chen, L. Kong, and L. Rao, “Auc2charge: An online auction framework for eectric vehicle park-and-charge,” in _Proceedings of the 2015 ACM Sixth International Conference on Future Energy Systems_ , ser. e-Energy ’15. New York, NY, USA: Association for Computing Machinery, 2015, p. 151–160.
  * [62] R. Zhou, Z. Li, C. Wu, and M. Chen, “Demand response in smart grids: A randomized auction approach,” _IEEE Journal on Selected Areas in Communications_ , vol. 33, no. 12, pp. 2540–2553, 2015.
  * [63] R. Zhou, Z. Li, and C. Wu, “An online procurement auction for power demand response in storage-assisted smart grids,” in _2015 IEEE Conference on Computer Communications (INFOCOM)_ , 2015, pp. 2641–2649.
  * [64] C.-K. Chau, K. Elbassioni, and M. Khonji, “Truthful mechanisms for combinatorial allocation of electric power in alternating current electric systems for smart grid,” _ACM Trans. Econ. Comput._ , vol. 5, no. 1, Oct. 2016.
  * [65] C. Akasiadis and G. Chalkiadakis, “Cooperative electricity consumption shifting,” _Sustainable Energy, Grids and Networks_ , vol. 9, pp. 38 – 58, 2017.
  * [66] C. Akasiadis and G. Chalkiadakis, “Mechanism design for demand-side management,” _IEEE Intelligent Systems_ , vol. 32, no. 1, pp. 24–31, 2017\.
  * [67] G. Yuan, C. Hang, M. N. Huhns, and M. P. Singh, “A mechanism for cooperative demand-side management,” in _2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS)_ , 2017, pp. 361–371.
  * [68] R. Meir, H. Ma, and V. Robu, “Contract design for energy demand response,” 2017\.
  * [69] W. Zhong, K. Xie, Y. Liu, C. Yang, and S. Xie, “Efficient auction mechanisms for two-layer vehicle-to-grid energy trading in smart grid,” in _2017 IEEE International Conference on Communications (ICC)_ , 2017, pp. 1–6.
  * [70] G. Methenitis, M. Kaisers, and H. La Poutré, “Forecast-based mechanisms for demand response.” in _AAMAS_ , 2019, pp. 1600–1608.
  * [71] M. Khorasany, Y. Mishra, and G. Ledwich, “Design of auction-based approach for market clearing in peer-to-peer market platform,” _The Journal of Engineering_ , vol. 2019, no. 18, pp. 4813–4818, 2019.
  * [72] A. Afzaal, F. Kanwal, A. H. Ali, K. Bashir, and F. Anjum, “Agent-based energy consumption scheduling for smart grids: An auction-theoretic approach,” _IEEE Access_ , vol. 8, pp. 73 780–73 790, 2020.
  * [73] M. K. AlAshery, Z. Yi, D. Shi, X. Lu, C. Xu, Z. Wang, and W. Qiao, “A blockchain-enabled multi-settlement quasi-ideal peer-to-peer trading framework,” _IEEE Trans. Smart Grid_ , vol. 12, no. 1, pp. 885–896, 2021.
  * [74] M. Roveto, R. Mieth, and Y. Dvorkin, “Co-optimization of var and cvar for data-driven stochastic demand response auction,” _IEEE Control Systems Letters_ , vol. 4, no. 4, pp. 940–945, 2020.
  * [75] D. Sypatayev, H. S. V. S. Kumar Nunna, and A. Shintemirov, “A novel peer-to-peer negawatt trading transactive energy system for prosumers,” in _2020 IEEE 14th International Conference on Compatibility, Power Electronics and Power Engineering (CPE-POWERENG)_ , vol. 1, 2020, pp. 181–186.
  * [76] X. Wei and A. Anastasopoulos, “Mechanism design for demand management in energy communities,” 2020.
  * [77] W. Zhong, K. Xie, Y. Liu, C. Yang, and S. Xie, “Multi-resource allocation of shared energy storage: A distributed combinatorial auction approach,” _IEEE Trans. Smart Grid_ , vol. 11, no. 5, pp. 4105–4115, 2020\.
  * [78] J. Kang, R. Yu, X. Huang, S. Maharjan, Y. Zhang, and E. Hossain, “Enabling localized peer-to-peer electricity trading among plug-in hybrid electric vehicles using consortium blockchains,” _IEEE Trans. Industrial Informatics_ , vol. 13, no. 6, pp. 3154–3164, 2017.
  * [79] Y. Zhao, X. Jia, Q. Yang, D. Li, and D. An, “Towards incentive compatible auction mechanism for electric vehicles bidding in microgrids,” in _2018 33rd Youth Academic Annual Conference of Chinese Association of Automation (YAC)_ , 2018, pp. 334–339.
  * [80] W. Zhong, C. Yang, K. Xie, S. Xie, and Y. Zhang, “Admm-based distributed auction mechanism for energy hub scheduling in smart buildings,” _IEEE Access_ , vol. 6, pp. 45 635–45 645, 2018.
  * [81] Y. Okajima, T. Murao, K. Hirata, and K. Uchida, “Integration mechanisms for lq energy day-ahead market based on demand response,” in _2014 IEEE Conference on Control Applications (CCA)_ , 2014, pp. 1–8.
  * [82] J. Xu and M. van der Schaar, “Incentive-compatible demand-side management for smart grids based on review strategies,” _EURASIP Journal on Advances in Signal Processing_ , vol. 2015, no. 1, p. 51, 2015.
  * [83] T. Murao, K. Hirata, Y. Okajima, and K. Uchida, “Real-time pricing for lqg power networks with independent types: A dynamic mechanism design approach,” _European Journal of Control_ , vol. 39, pp. 95 – 105, 2018.
  * [84] S. Chakraborty, T. Ito, and T. Senjyu, “Smart pricing scheme: A multi-layered scoring rule application,” _Expert Systems with Applications_ , vol. 41, no. 8, pp. 3726 – 3735, 2014.
  * [85] S. Jain, B. Narayanaswamy, and Y. Narahari, “A multiarmed bandit incentive mechanism for crowdsourcing demand response in smart grids,” _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 28, no. 1, Jun. 2014.
  * [86] S. Chakraborty and T. Ito, _Hierarchical Scoring Rule Based Smart Dynamic Electricity Pricing Scheme_. Tokyo: Springer Japan, 2015, pp. 113–131.
  * [87] S. Mhanna, G. Verbič, and A. C. Chapman, “A faithful distributed mechanism for demand response aggregation,” _IEEE Trans. Smart Grid_ , vol. 7, no. 3, pp. 1743–1753, 2016.
  * [88] A. C. Chapman, G. Verbič, and D. J. Hill, “Algorithmic and strategic aspects to integrating demand-side aggregation and energy management methods,” _IEEE Trans. Smart Grid_ , vol. 7, no. 6, pp. 2748–2760, 2016\.
  * [89] P. Ströhle and C. M. Flath, “Local matching of flexible load in smart grids,” _European Journal of Operational Research_ , vol. 253, no. 3, pp. 811 – 824, 2016.
  * [90] A. Ehsanfar and B. Heydari, “An incentive-compatible scheme for electricity cooperatives: An axiomatic approach,” _IEEE Trans. Smart Grid_ , vol. 9, no. 2, pp. 1416–1424, 2018.
  * [91] J. Vuelvas, F. Ruiz, and G. Gruosso, “Limiting gaming opportunities on incentive-based demand response programs,” _Applied Energy_ , vol. 225, pp. 668 – 681, 2018.
  * [92] J. Vuelvas, F. Ruiz, and G. Gruosso, “A contract for demand response based on probability of call,” in _2018 IEEE International Conference on Industrial Technology (ICIT)_ , 2018, pp. 1095–1100.
  * [93] N. Aizenberg, E. Stashkevich, and N. Voropai, “Forming rate options for various types of consumers in the retail electricity market by solving the adverse selection problem,” _International Journal of Public Administration_ , vol. 42, no. 15-16, pp. 1349–1362, 2019.
  * [94] D. Muthirayan, D. Kalathil, K. Poolla, and P. Varaiya, “Baseline estimation and scheduling for demand response,” in _2018 IEEE Conference on Decision and Control (CDC)_ , 2018, pp. 4857–4862.
  * [95] J. Shweta and G. Sujit, “A multiarmed bandit based incentive mechanism for a subset selection of customers for demand response in smart grids,” _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 02, pp. 2046–2053, Apr. 2020.
  * [96] D. Muthirayan, D. Kalathil, K. Poolla, and P. Varaiya, “Mechanism design for demand response programs,” _IEEE Trans. Smart Grid_ , vol. 11, no. 1, pp. 61–73, 2020.
  * [97] M. N. Faqiry and S. Das, “Double-sided energy auction in microgrid: Equilibrium under price anticipation,” _IEEE Access_ , vol. 4, pp. 3794–3805, 2016.
  * [98] S. Mhanna, A. C. Chapman, and G. Verbič, “A faithful and tractable distributed mechanism for residential electricity pricing,” _IEEE Trans. Power Systems_ , vol. 33, no. 4, pp. 4238–4252, 2018.
  * [99] Z. Zhou, B. Wang, W. Liao, and G. Xu, “Contract-based incentive-compatible demand response for internet of electric vehicles,” in _2018 2nd IEEE Conference on Energy Internet and Energy System Integration (EI2)_ , 2018, pp. 1–5.
  * [100] B. P. Majumder, M. N. Faqiry, S. Das, and A. Pahwa, “An efficient iterative double auction for energy trading in microgrids,” in _2014 IEEE Symposium on Computational Intelligence Applications in Smart Grid (CIASG)_ , 2014, pp. 1–7.
  * [101] M. N. Faqiry and S. Das, “A budget balanced energy distribution mechanism among consumers and prosumers in microgrid,” in _2016 IEEE International Conference on Internet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData)_ , 2016, pp. 516–520.
  * [102] ——, “Double auction with hidden user information: Application to energy transaction in microgrid,” _IEEE Trans. Systems, Man, and Cybernetics: Systems_ , vol. 49, no. 11, pp. 2326–2339, 2019.
  * [103] ——, “Distributed bilevel energy allocation mechanism with grid constraints and hidden user information,” _IEEE Trans. Smart Grid_ , vol. 10, no. 2, pp. 1869–1879, 2019.
  * [104] N. Li, L. Chen, and S. H. Low, “Optimal demand response based on utility maximization in power networks,” in _2011 IEEE Power and Energy Society General Meeting_ , 2011, pp. 1–8.
  * [105] M. H. K. Tushar, C. Assi, and M. Maier, “Distributed real-time electricity allocation mechanism for large residential microgrid,” _IEEE Trans. Smart Grid_ , vol. 6, no. 3, pp. 1353–1363, 2015.
  * [106] T. Namerikawa, N. Okubo, R. Sato, Y. Okawa, and M. Ono, “Real-time pricing mechanism for electricity market with built-in incentive for participation,” _IEEE Trans. Smart Grid_ , vol. 6, no. 6, pp. 2714–2724, 2015.
  * [107] A. C. Chapman and G. Verbič, “An iterative on-line auction mechanism for aggregated demand-side participation,” _IEEE Trans. Smart Grid_ , vol. 8, no. 1, pp. 158–168, 2017.
  * [108] K. Steriotis, G. Tsaousoglou, N. Efthymiopoulos, P. Makris, and E. M. Varvarigos, “A novel behavioral real time pricing scheme for the active energy consumers’ participation in emerging flexibility markets,” _Sustainable Energy, Grids and Networks_ , vol. 16, pp. 14 – 27, 2018.
  * [109] Y. Zhou, S. Ci, H. Li, and Y. Yang, “Designing pricing incentive mechanism for proactive demand response in smart grid,” in _2018 IEEE International Conference on Communications (ICC)_ , 2018, pp. 1–6.
  * [110] G. Tsaousoglou, K. Steriotis, N. Efthymiopoulos, K. Smpoukis, and E. Varvarigos, “Near-optimal demand side management for retail electricity markets with strategic users and coupling constraints,” _Sustainable Energy, Grids and Networks_ , vol. 19, p. 100236, 2019.
  * [111] G. Tsaousoglou, N. Efthymiopoulos, P. Makris, and E. Varv Arigos, “Personalized real time pricing for efficient and fair demand response in energy cooperatives and highly competitive flexibility markets,” _Journal of Modern Power Systems and Clean Energy_ , vol. 7, no. 1, pp. 151–162, 2019.
  * [112] C. Barreto, E. Mojica-Nava, and N. Quijano, “Incentive mechanisms to prevent efficiency loss of non-profit utilities,” _International Journal of Electrical Power & Energy Systems_, vol. 110, pp. 523 – 535, 2019.
  * [113] L. Hou, S. Ma, J. Yan, C. Wang, and J. Y. Yu, “Reinforcement mechanism design for electric vehicle demand response in microgrid charging stations,” in _2020 International Joint Conference on Neural Networks (IJCNN)_ , 2020, pp. 1–8.
  * [114] M. Latifi, A. Khalili, A. Rastegarnia, W. M. Bazzi, and S. Sanei, “A robust scalable demand-side management based on diffusion-admm strategy for smart grid,” _IEEE Internet of Things Journal_ , vol. 7, no. 4, pp. 3363–3377, 2020.
  * [115] T. Namerikawa and Y. Okawa, _Distributed Dynamic Pricing in Electricity Market with Information Privacy_. Singapore: Springer Singapore, 2020, pp. 213–244.
  * [116] L. Wang, J. Chen, S. Zeng, L. Liu, and K. Peng, “Reward-punishment based user utility maximization model for optimal real-time pricing in electricity energy supply,” in _2020 IEEE Power Energy Society Innovative Smart Grid Technologies Conference (ISGT)_ , 2020, pp. 1–5.
  * [117] E. Bitar and Y. Xu, “On incentive compatibility of deadline differentiated pricing for deferrable demand,” in _52nd IEEE Conference on Decision and Control_ , 2013, pp. 5620–5627.
  * [118] C. Barreto, E. Mojica-Nava, and N. Quijano, “Design of mechanisms for demand response programs,” in _52nd IEEE Conference on Decision and Control_ , 2013, pp. 1828–1833.
  * [119] C. Barreto and A. A. Cárdenas, “Incentives for demand-response programs with nonlinear, piece-wise continuous electricity cost functions,” in _2015 American Control Conference (ACC)_ , 2015, pp. 4327–4332.
  * [120] E. Bitar and Y. Xu, “Deadline differentiated pricing of deferrable electric loads,” _IEEE Trans. Smart Grid_ , vol. 8, no. 1, pp. 13–25, 2017\.
  * [121] X. Tan, A. Leon-Garcia, Y. Wu, and D. H. K. Tsang, “Posted-price retailing of transactive energy: An optimal online mechanism without prediction,” _IEEE Journal on Selected Areas in Communications_ , vol. 38, no. 1, pp. 5–16, 2020.
  * [122] A. Gibbard, “Manipulation of voting schemes: A general result,” _Econometrica_ , vol. 41, no. 4, pp. 587–601, 1973.
  * [123] R. B. Myerson, “Incentive compatibility and the bargaining problem,” _Econometrica_ , vol. 47, no. 1, pp. 61–73, 1979.
  * [124] Y. Bachrach, P. Key, and M. Zadimoghaddam, “Collusion in vcg path procurement auctions,” in _Internet and Network Economics_ , A. Saberi, Ed. Berlin, Heidelberg: Springer Berlin Heidelberg, 2010, pp. 38–49.
  * [125] O. Morgenstern and J. Von Neumann, _Theory of games and economic behavior_. Princeton university press, 1953\.
  * [126] K. M. Ormazabal, “The law of diminishing marginal utility in alfred marshall’s principles of economics,” _The European Journal of the History of Economic Thought_ , vol. 2, no. 1, pp. 91–126, 1995.


# Maha Bhaashya at SemEval-2024 Task 6: Zero-Shot Multi-task Hallucination
Detection

Patanjali Bhamidipati†
IIIT Hyderabad
<EMAIL_ADDRESS>
Advaith Malladi†
IIIT Hyderabad
<EMAIL_ADDRESS>Manish Shrivastava
IIIT Hyderabad
<EMAIL_ADDRESS>
Radhika Mamidi
IIIT Hyderabad
<EMAIL_ADDRESS>
###### Abstract

In recent studies, the extensive utilization of large language models has
underscored the importance of robust evaluation methodologies for assessing
text generation quality and relevance to specific tasks. This has revealed a
prevalent issue known as hallucination, an emergent condition in the model
where generated text lacks faithfulness to the source and deviates from the
evaluation criteria. In this study, we formally define hallucination and
propose a framework for its quantitative detection in a zero-shot setting,
leveraging our definition and the assumption that model outputs entail task
and sample specific inputs. In detecting hallucinations, our solution achieves
an accuracy of 0.78 in a model-aware setting and 0.61 in a model-agnostic
setting. Notably, our solution maintains computational efficiency, requiring
far less computational resources than other SOTA approaches, aligning with the
trend towards lightweight and compressed models.

Maha Bhaashya at SemEval-2024 Task 6: Zero-Shot Multi-task Hallucination
Detection

Patanjali Bhamidipati† IIIT Hyderabad<EMAIL_ADDRESS>Advaith
Malladi† IIIT Hyderabad<EMAIL_ADDRESS>

Manish Shrivastava IIIT Hyderabad<EMAIL_ADDRESS>Radhika Mamidi IIIT
Hyderabad<EMAIL_ADDRESS>

$\dagger$$\dagger$footnotetext: The authors contributed equally to this work.

##  1 Introduction

The contemporary landscape of Natural Language Generation (NLG) is marked by a
confluence of complexities, wherein two primary challenges emerge as focal
points of concern. Firstly, the prevalent neural models within NLG frameworks
consistently produce outputs that exhibit linguistic fluency yet suffer from
inaccuracies. Secondly, the current evaluation metrics, vital for evaluating
the effectiveness of NLG systems, demonstrate a significant inclination
towards fluency measures while neglecting to prioritize accuracy. So, this
highlights the need to consider the "truthfulness" of the model’s output, i.e
its alignment with the source to ensure a comprehensive assessment.Dale et al.
(2022)

In the realm of NLG applications, the criticality of output accuracy cannot be
overstated. A divergence between the fluency and factual correctness of
generated content not only undermines the utility of NLG systems but also
engenders substantial risks across various domains. Consider, for instance,
the domain of machine translation,the production of seemingly plausible yet
inaccurate translations not only compromises the integrity of the translated
content but also defeats the purpose of facilitating correct translations.

Likewise, in tasks like definition modeling and paraphrase generation, where
accurately conveying semantic meaning is crucial, the presence of incorrect
outputs presents notable challenges in upholding the integrity and
dependability of the generated content.

##  2 SHROOM Dataset

SHROOM (a Shared-task on Hallucinations and Related Observable Overgeneration
Mistakes) dataset is a task-based hallucination detection dataset which is
divided into two major categories:

###  2.1 Model Aware (MAw)

Model Aware (MAw) refers to situations where the model under study is known.

###  2.2 Model Agnostic (MAg)

Model Agnostic (MAg) refers to situations where the model under study is not
known.

The dataset encompasses three major Natural Language Generation tasks, namely:

1\. Definition Modeling (DM): In this task, models are trained to generate a
definition for a given example in context.

2\. Machine Translation (MT): In this task, models aim to generate
translations of the given samples.

3\. Paraphrase Generation (PG): In this task, models aim to produce
paraphrases of the given text samples.

Further, each sample in the train set is populated with information such as
task ($task$): indicating what objective the model was trained for, source
($src$): the input passed to the models for the generation, target ($tgt$):
the intended reference "gold" standard text that the model ought to generate,
hypothesis ($hyp$): the actual model production,also the model-aware dataset
is populated with model name ($model$) used for the task, with the val set
additionally being populated with majority-based gold-label ($label$), based
on the annotator labels along with the probability values of the sample being
hallucination ($p(Hallucination)$) based on the proportion of annotators who
claim that the sample is an hallucination.

##  3 Definitions

As described earlier, the SHROOM shared task encompasses of three different
taks, Definition Modelling (DM), Paraphrase Generation (PG), and Machine
Translation. We define the Hallucination in the context of the specific task
at hand. Defining hallucinations individually in the context of a specific
task enables detecting hallucinations quantitatively and qualitatively. We
offer distinct definitions of hallucinations and methodologies for detecting
hallucinations within the context of each of the aforementioned task.

In the context of definition modelling, the model is expected to generate the
definition of a word which has been used in the provided context by making use
of distributional semantics. Definition modelling models such as
flan-t5-definition-en-base Giulianelli et al. (2023) are not fully capable of
making use of distributional semantics to define a word as used in a context.
In a sample where the word W has been used in a setting contrasting to the
definition the model has learnt during its training process, the models fails
to provide a contextual definition of the word W. Examples for the same have
been demonstrated in Table 1 and Table 2. We observe the model outputs a
definition of word W which is very similar what it has learnt during its
training process. Based on this observation, we assume that the targets
provided in the SHROOM dataset have been extracted from the training data of
the definition modelling dataset. With this assumption, we define
"Hallucination to be an instance where the output generated by the definition
modelling model does not entail the target output." Thereby reducing the
hallucination detection task to a Natural Language Inference task in the
context of definition modelling.

In the context of paraphrase generation and machine translation, the model’s
inputs and outputs are anticipated to exhibit semantic equivalence. If the
generated paraphrase or translation diverges from semantic equivalence with
the source text, they are deemed imperfect paraphrases or translations.
Therefore, in the context of paraphrase generation and machine translation, we
define "Hallucination to be an instance where the paraphrase or translation
generated by the model is not semantically equivalent to the source."  This
reduces the hallucination detection in the context of given tasks to a
semantic equivalence detection task, which could also be framed as
bidirectional entailment detection, a variation of the Natural Language
Inference task.

The aforementioned definitions of hallucination allow us to simplify the
hallucination detection task to a Natural Language Inference task, thereby
enabling us to qualitatively and quantitatively detect hallucinations.

In a more generic setting, we provide a definition of hallucinations that can
be adapted to any task to effectively detect them. We define "hallucinations
as instances where the output generated by the model is not faithful to the
input or the training data of the model. If the model generates information
that is contradictory to the model’s training data or the input to the model,
it can be termed as a hallucination."

Example 1: Definition Modeling (DM)
---
Model Input: I went into the water bottle to withdraw cash. What is the
definition of water bottle?
Model Output: A container for holding liquids.
Expected Output: A financial institution such as a bank or ATM to withdraw
cash
Model: flan-t5-definition-en-base
Table 1:  A Table showing distinction between the model output and the
expected output where the model fails to understand the contextual definition
of the term water bottle Example 2: Definition Modeling (DM)
---
I jumped into the flaxcron to do some swimming. What is the meaning of
flaxcron?
Model Output: A slender, slender
Expected Output: A pool of water.
Model: flan-t5-definition-en-base
Table 2:  A Table showing distinction between the model output and the
expected output where the model fails to understand the contextual definition
of the term flaxcron

##  4 Methodology

Grounding to the above definitions, the experimental setup we designed goes on
to quantify the alignment of the model’s output ($hyp$) with either the source
($src$) or the target ($tgt$) based on the task ($task$) the data sample
corresponds to.

We propose that examining the entailment relationship between the model’s
output ($hyp$) and either the source ($src$) or the target ($tgt$) (which is
also inherently linked to the source ($src$)), depending on the task, sheds
light on data samples that are not "detached" from the source. Consistent with
our initial hypothesis that hallucinations occur when samples are "detached"
from the source, this approach based on Natural Language Inference (NLI) can
effectively aid in hallucination classification.

  * •

In the context of definition modelling, if the $hyp$ does not entail the
$tgt$, the sample has been classified as Hallucination.

  * •

In the context of machine translation and paraphrase generation, we check
equivalence through bidirectional entailment. If the $hyp$ does not
bidirectionally entail the $src$, the sample has been classified as
Hallucination

  * •

In the context of machine translation and paraphrase generation, we verify our
hypothesis of semantic equivalence between the $src$ and $hyp$ by comparing
the performance metrics in the case of both unidirectional and bidirectional
entailment.

Recent research heavily relies on large language models (LLMs) to benchmark
various natural language understanding and generation (NLG) tasks. However,
this practice extends to hallucination detection as well, which we find ironic
and counterproductive, considering LLMs’ inherent tendency to hallucinate.
Using LLMs for hallucination detection presents two major drawbacks. Firstly,
their computational demands are significant, making them an expensive solution
Bai et al. (2024). Secondly, the lack of complete interpretability in LLMs
renders them unreliable for this task Singh et al. (2024).

##  5 Results

After several experiments with the above methodology, leveraging the accuracy
and the Spearman correlation ($\rho$) metrics, we have bench-marked the
hallucination detection task on the SHROOM validation and test sets to achieve
an accuracy of 0.78 in model-aware and 0.61 on model-agnostic test sets
respectively. For our analysis let us take only the accuracy metric into
account.
The bench-marking saw a utilisation of open-source pre-trained Natural
Language Inference (NLI) models available on Hugging Face. Several experiments
brought out interesting observations which are worthy discussing.

We evaluated the following models:

  1. 1.

MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli (DeBERTa-1) He et al. (2020)

  2. 2.

MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7 (DeBERTa-2) He et
al. (2020)

  3. 3.

ynie/bart-large-snli_mnli_fever_anli_R1_R2_R3-nli (BART-1) Lewis et al. (2019)

  4. 4.

ynie/roberta-large-snli_mnli_fever_anli_R1_R2_R3-nli (RoBERTa-1) Nie et al.
(2020)

###  5.1 Definition Modelling

For the task of definition modelling, our approach achieves a peak accuracy of
0.748663 using the DeBERTa-2 model in a model-agnostic setting and a peak
accuracy of 0.755319 using the RoBERTa-1 model in a model-aware setting. These
results are in accordance with our hypothesis that when the model
hallucinates, it does not entail the target.

Model | Unidirectional | Bidirectional
---|---|---
DeBERTa - 1 | 0.783567 | 0.755511
DeBERTa - 2 | 0.765531 | 0.717435
BART - 1 | 0.769539 | 0.733467
RoBERTa - 1 | 0.757515 | 0.727455
Table 3:  Model-agnostic evaluation on (Uni vs Bi) directional entailment.  Model | Unidirectional | Bidirectional
---|---|---
DeBERTa - 1 | 0.596806 | 0.570859
DeBERTa - 2 | 0.576846 | 0.586826
BART - 1 | 0.610778 | 0.568862
RoBERTa - 1 | 0.612774 | 0.584830
Table 4:  Model-aware evaluation on (Uni vs Bi) directional entailment.  Model | Unidirectional | Bidirectional
---|---|---
DeBERTa - 1 | 0.728 | 0.752
DeBERTa - 2 | 0.624 | 0.68
BART - 1 | 0.696 | 0.72
RoBERTa - 1 | 0.712 | 0.752
Table 5:  Accuracy validation on PG task  Model | Unidirectional | Bidirectional
---|---|---
DeBERTa - 1 | - | -
DeBERTa - 2 | 0.722 | 0.754
BART - 1 | - | -
RoBERTa - 1 | - | -
Table 6:  Accuracy validation on MT task.  Model | DM | MT | PG
---|---|---|---
DeBERTa - 1 | 0.721925 | 0.855615 | 0.768
DeBERTa - 2 | 0.748663 | 0.823529 | 0.704
BART - 1 | 0.748663 | 0.844920 | 0.688
RoBERTa - 1 | 0.711230 | 0.834224 | 0.712
Table 7:  Model-agnostic evaluation on individual tasks.  Model | DM | MT | PG
---|---|---|---
DeBERTa - 1 | 0.744681 | 0.707447 | 0.208
DeBERTa - 2 | 0.670213 | 0.718085 | 0.224
BART - 1 | 0.728723 | 0.760638 | 0.208
RoBERTa - 1 | 0.755319 | 0.734042 | 0.216
Table 8:  Model-aware evaluation on individual tasks.

###  5.2 Paraphrase Generation and Machine Translation

For the paraphrase generation and machine translation tasks, the observed
results confirm our hypothesis that if the source (src) and hypothesis (hyp)
are not semantically equivalent, the hypothesis is a hallucination. In
hallucination detection for the paraphrase generation task, we observe that
bidirectional entailment outperforms the unidirectional entailment approach
for all models. Similar results can also be observed for the machine
translation task. This provides evidence that in machine translation and
paraphrase generation, hallucinations can be detected by checking for semantic
equivalency between the source and hypothesis.

###  5.3 Overall Analysis

In the model-agnostic setting, we achieve a peak accuracy of 0.783567 using
the DeBERTa-1 model and a peak accuracy of 0.612774 in a model-aware setting
using the RoBERTa-1 model. These scores exhibit satisfactory performance of
models pre-trained on the Natural Language Inference task for Hallucination
Detection.

##  6 Conclusion

Our work makes two significant contributions to the study of hallucinations in
language models. First, we provide a concrete definition of the term
"hallucination," enabling both qualitative and quantitative study and
detection of such phenomena. Second, we offer a computationally efficient
approach to detect hallucinations in tasks such as definition modeling,
machine translation, and paraphrase generation. We frame the hallucination
detection task as a function of the input to the generation model and the data
used to train it. Our definitions and approaches also provide a framework that
can be utilized for hallucination detection in various Natural Language
Generation tasks across the spectrum.

## References

  * Bai et al. (2024) Guangji Bai, Zheng Chai, Chen Ling, Shiyu Wang, Jiaying Lu, Nan Zhang, Tingwei Shi, Ziyang Yu, Mengdan Zhu, Yifei Zhang, Carl Yang, Yue Cheng, and Liang Zhao. 2024.  Beyond efficiency: A systematic survey of resource-efficient large language models.
  * Dale et al. (2022) David Dale, Elena Voita, Loïc Barrault, and Marta R. Costa-jussà. 2022.  Detecting and mitigating hallucinations in machine translation: Model internal workings alone do well, sentence similarity even better.
  * Giulianelli et al. (2023) Mario Giulianelli, Iris Luden, Raquel Fernandez, and Andrey Kutuzov. 2023.  Interpretable word sense representations via definition generation: The case of semantic change analysis.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3130–3148, Toronto, Canada. Association for Computational Linguistics.
  * He et al. (2020) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020.  Deberta: Decoding-enhanced BERT with disentangled attention.  _CoRR_ , abs/2006.03654.
  * Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2019.  BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _CoRR_ , abs/1910.13461.
  * Nie et al. (2020) Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020.  Adversarial NLI: A new benchmark for natural language understanding.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics.
  * Singh et al. (2024) Chandan Singh, Jeevana Priya Inala, Michel Galley, Rich Caruana, and Jianfeng Gao. 2024.  Rethinking interpretability in the era of large language models.