# Attentive Convolutional Deep Reinforcement Learning for Optimizing Solar-
Storage Systems in Real-Time Electricity Markets

Jinhao Li  Changlong Wang  Hao Wang This work was supported in part by the
Australian Research Council (ARC) Discovery Early Career Researcher Award
(DECRA) under Grant DE230100046. (Corresponding author: Hao Wang.)J. Li and H.
Wang are with the Department of Data Science and AI, Faculty of IT and Monash
Energy Institute, Monash University, Melbourne, VIC 3800, Australia (e-mail:
{jinhao.li,hao.wang2}@monash.edu).C. Wang is with the Department of Civil
Engineering, Monash University, Melbourne, VIC 3800, Australia (e-mail:
chang.wang@monash.edu).

###### Abstract

This paper studies the synergy of solar-battery energy storage system (BESS)
and develops a viable strategy for the BESS to unlock its economic potential
by serving as a backup to reduce solar curtailments while also participating
in the electricity market. We model the real-time bidding of the solar-battery
system as two Markov decision processes for the solar farm and the BESS,
respectively. We develop a novel deep reinforcement learning (DRL) algorithm
to solve the problem by leveraging attention mechanism (AC) and multi-grained
feature convolution to process DRL input for better bidding decisions.
Simulation results demonstrate that our AC-DRL outperforms two optimization-
based and one DRL-based benchmarks by generating $23\%$, $20\%$, and $11\%$
higher revenue, as well as improving curtailment responses. The excess solar
generation can effectively charge the BESS to bid in the market, significantly
reducing solar curtailments by $76\%$ and creating synergy for the solar-
battery system to be more viable.

Solar photovoltaic, solar curtailment, battery energy storage system, deep
reinforcement learning, electricity market.

##  1 Introduction

###  1.1 Background and Literature Review

Solar photovoltaic (PV) has attracted nearly half of the global renewable
investment and become the leading source of utility-scale renewables [1].
However, due to the inherent variability of solar PV generation, curtailment
of solar PV outputs is often inevitable but causes significant economic losses
for solar farms [2]. Therefore, effective efforts for solar curtailment
mitigation are urgent for power system operators as a key factor for the
successful grid transition, as well as for solar farm owners for profitability
concerns.

The grid-scale battery energy storage system (BESS) has emerged as a critical
solution for reducing solar curtailments and improving the economic
performance of solar farms [3]. In co-located solar-battery systems, the BESS
usually serves as the onsite backup source to smooth the fluctuating solar
generation by absorbing excess solar energy (which is otherwise curtailed) and
releasing stored energy when the solar PV generation is insufficient. However,
the limited ancillary role fails to unlock the BESS’s economic potential and
cannot justify the economic viability of co-located BESS, due to the high
upfront cost of the BESS despite its declining cost over the past decades. To
enhance the economic viability, the BESS can take advantage of its operational
flexibility to perform energy arbitrage (i.e., buy low and sell high) in the
wholesale electricity market as a price-taker, creating an additional revenue
stream for the BESS complementing its ancillary role in reducing solar
curtailment. Therefore, it is of great significance to design an effective
coordination strategy for the co-located solar-battery system to concurrently
reduce solar curtailment and enhance the BESS’s viability, creating a win-win
situation for solar farms, the BESS, and the power system. However, designing
such an effective coordination strategy is not trivial but challenging due to
the uncertainty of solar energy, volatility of the energy market, and BESS
decision coupling between curtailment management and energy arbitrage.

Optimization-based methods have been introduced to optimize bidding decisions
of the co-located solar-battery system in the electricity market. To tackle
the uncertainties of solar generation and the electricity market, stochastic
optimization (SO) [4] and robust optimization (RO) [5] have been introduced to
maximize the overall revenue of the solar-battery system, but often overlook
the importance of solar curtailment, thereby failing to take advantage of
curtailed solar energy to charge the BESS without additional fees. In
addition, the above works primarily focused on the day-ahead electricity
market, i.e., scheduling the BESS’s operations for the next $24$ hours. In
contrast to the day-ahead market, the real-time electricity market presents
higher profitability than the day-ahead market [6], but also greater
challenges due to the high volatility of the real-time market and complex
price drivers.

Optimization approaches for real-time bidding, in particular the model
predictive control (MPC) and its variants, have been widely studied in the
literature. For example, a deterministic MPC (DMPC) method has been proposed
by [7] for a BESS taking part in real-time electricity market. Stochastic MPC
(SMPC) [8], robust MPC (RMPC) [9], and ensemble nonlinear MPC (EnNMPC) [10]
methods have also been introduced to further characterize and capture the
uncertainties of both solar generation and electricity price. However, the
aforementioned four kinds of MPC-based methods also depend on the quality of
forecast results, as the predictions of solar generation and prices are the
parameters for the optimization solver, thereby affecting the optimality of
bidding decisions. Additionally, the electricity price is notoriously
challenging to predict [11].

Besides MPC-based approaches, deep reinforcement learning (DRL) has drawn
increasing attention in real-time decision-making problems, for its powerful
capability to directly learn the dynamics of the environment (such as solar
generation and the electricity market) and consequently capture the
uncertainties of time-varying parameters without using explicit forecast
models. Wang et al. [12] managed a multi-energy microgrid via a DRL-based
approach, while they operated the BESS as an ancillary asset. One recent study
[13] discussed the real-time bidding of a co-located solar-battery system.
However, the integrated system functioned as a single participant in real-time
markets, ignoring the underlying coordination between the two facilities.

The synergy of a co-located wind-battery system in real-time markets has been
studied in [14], but one essential factor was neglected. How to effectively
use available information as features for deep reinforcement learning (DRL) is
not well understood, but it is essential to DRL-based bidding decision-making.
For example, the BESS’s arbitrage decision is influenced by the energy price,
and energy exchange in the market is impacted by the physical battery
capacity. Existing studies have not addressed this question, thus highlighting
the need for research on effectively harnessing input features, including
solar generation, battery capacity, and energy price, to support DRL-based
bidding strategies.

###  1.2 Contributions and Paper Structure

Motivated by the need for effective coordination strategies for the solar-
battery system, encompassing curtailment mitigation and market participation
(i.e., real-time energy arbitrage), we leverage the data-driven ability of DRL
to develop a novel DRL-based bidding strategy for co-located solar-battery
systems. Our strategy aims to concurrently manage solar curtailment and
optimize the system’s participation in the wholesale real-time market. By
employing DRL, we can learn the uncertainties of solar power and energy prices
in a model-free manner and derive the coordination strategy from historical
experiences through interactive training without relying on prior knowledge or
forecasts. Furthermore, to better explore how various input features influence
the bidding decision-making for both the solar farm and the BESS, we propose a
novel DRL network structure comprising attention mechanism and multi-grained
feature convolution. Specifically, the attention mechanism can exploit the
correlations among features and emphasize the relative significance of each
feature, which is then processed by multi-grained feature convolution to make
bidding decisions. This structure enables both the BESS and the solar farm to
focus on different input features during the dynamic bidding process. We refer
to our method as AC-DRL (Attentive Convolutional DRL). The main contributions
of this paper are summarized as follows.

  * •

Optimizing Coordination of Solar Curtailment Reduction and BESS Bidding in
Real-Time Electricity Market: We study coordinated solar-battery operations to
simultaneously manage solar curtailment and perform energy arbitrage in a
real-time market, creating a more promising revenue stream than purely
operating as onsite backup sources. Our work provides a viable case for the
co-location of renewable generators and the BESS by demonstrating the synergy
of the co-located solar-battery system to unlock its economic potential.

  * •

Attentive Convolutional DRL-based Bidding: We design a novel DRL network
structure, namely AC-DRL, incorporating a stacked attention mechanism and a
multi-grained feature convolution module. Our AC-DRL fully explores the
correlations of DRL input and effectively identifies the relative significance
of input features, resulting in better and more informed decision-making. With
the AC-DRL, we further decouple the market participation of the co-located
system into two correlated Markov decision processes, enabling us to better
analyze how multiple factors, e.g., energy price and solar generation, affect
the decision-making of the co-located system.

  * •

Simulations and Insights Based on Real-World Data: Using realistic solar farm
data collected from the Australian National Electricity Market (NEM), our
simulations demonstrate the effectiveness of our AC-DRL method. The results
show that AC-DRL significantly outperforms both MPC-based and DRL-based
benchmarks. It is revealed that effective solar curtailment management is the
key to the successful coordination of the two assets, improving energy
arbitrage and enhancing the economic viability of the co-located solar-battery
system.

The remainder of this paper is organized as follows. Section 2 introduces the
solar-battery system and formulates the coordination problem of its
participation in real-time market. Section 3 disentangles the system’s bidding
process and presents our AC-DRL framework to optimize the overall revenue
while reducing solar curtailments. Section 4 presents simulation results, and
Section 5 concludes this paper.

Figure 1: The system model paradigm.

##  2 System Model

We consider a co-located solar-battery system participating as a price-taker
in real-time spot market (Fig. 1). This assumption is reasonable for small
systems with negligible market power, as distributed solar and storage
resources proliferate. Our system size is minor compared to the significant
NEM demand [15]. The price-taker focus enables studying the key problem of
solar-battery coordination and bidding agent design, while isolating broader
market effects. It has precedent in DRL [16] and optimization [4] bidding
literature. Although simplified compared to full multi-agent interactions, it
allows initial benchmarking of bidding performance. Nevertheless, there is
still an export limit for the co-located system to the grid, which is assumed
to be $62.5\%$ of the solar-battery system’s installed capacity, as described
in [17]. In the co-located solar-battery system, the BESS simultaneously
performs energy arbitrage in the spot market and charges otherwise curtailed
solar energy from the onsite solar farm during curtailment events. This
section is organized as follows. The background on the spot market is
introduced in Section 2.1. In Section 2.2, we model the solar farm and the
BESS revenue streams under various operational conditions. The overall bidding
problem of the solar-battery system is formulated in Section 2.3.

###  2.1 Background of the Spot Market

The spot market, as the main component of the NEM, functions as a real-time
market for wholesale electricity transactions, where the power mismatches
between generators and loads are immediately balanced by the Australian Energy
Market Operator (AEMO) through a centrally coordinated dispatch process [15].
Specifically, AEMO receives bids from generators every five minutes, and
dispatches generators in a cost-effective manner by ranking their bids from
low to high, forming a bidding stack. The bid price that meets the final
demand determines the market clearing price, known as the spot price (in
AU$/MWh) [15]. Our solar-battery system, like generators, once dispatched,
will be paid at the spot price.

###  2.2 The Solar-Battery System and Revenue Streams

####  2.2.1 Solar Farm

Due to the uncertainty of solar generation, most solar farms in the NEM
register as semi-scheduled generators. The AEMO requires the solar farm to
constantly update its plant availability (also known as the upper power
limit), denoted as $p_{t}^{\text{S,Ava}}$ from onsite monitoring devices [18],
based on which the solar farm can schedule a dispatch target, i.e., the amount
of bid power (in MW) in the spot market, denoted as $p_{t}^{\text{S}}$, to
fulfill in the next dispatch interval.

The intermittence of solar PV power often results in deviations between the
dispatch target and the actual solar generation, denoted as
$p_{t}^{\text{S,Act}}$. The deviation can make it challenging for the solar
farm to deliver the exact amount of dispatch target. For instance, if there is
a solar shortage, i.e., $p_{t}^{\text{S,Act}}<p_{t}^{\text{S}}$, the dispatch
target can only be partially met, while the excessive solar output will be
curtailed to match the dispatch target if there is excess solar generation,
i.e., $p_{t}^{\text{S,Act}}>p_{t}^{\text{S}}$. Therefore, the actual
dispatched solar power can be defined as
$p_{t}^{\text{S,Dis}}=\min\\{p_{t}^{\text{S,Act}},p_{t}^{\text{S}}\\}$.
Moreover, to regulate the bidding behaviors of the solar farm to comply with
the market rules [18], we introduce a penalty term if the solar farm does not
meet the dispatch target. We define $\lambda_{t}$ as the spot price, $T$ as
the overall time frame, $\Delta t$ as the NEM dispatch interval (in $5$-minute
resolution), and $\alpha$ as the aforementioned penalty coefficient [9]. The
revenue stream of the solar farm from the spot market can thus be formulated
as

$R^{\text{S}}=\Delta
t\sum_{t=1}^{T}\lambda_{t}\left(p_{t}^{\text{S,Dis}}-\alpha|p_{t}^{\text{S,Dis}}-p_{t}^{\text{S}}|\right).$
(1)

####  2.2.2 BESS

Price fluctuations in the spot market serve as a reflection of its inherent
volatility, where the lack of generation or demand leads to an increase or
decrease in prices, respectively. The BESS can take advantage of both the
market stochasticity and its operational flexibility by switching between its
two working modes (storage and generation) to buy energy at low prices and
sell it at high prices, known as energy arbitrage in the spot market. Given
that the BESS cannot simultaneously charge and discharge, we introduce two
binary variables, $v_{t}^{\text{Ch}}$ and $v_{t}^{\text{Dch}}$, to indicate
the BESS’s operational mode, leading to the following constraints

$v_{t}^{\text{Dch}}+v_{t}^{\text{Ch}}\leq 1,\leavevmode\nobreak\
v_{t}^{\text{Dch}},v_{t}^{\text{Ch}}\in\\{0,1\\}.$ (2)

The BESS sits idle when both binary variables are zero.

Let $p_{t}^{\text{Bat,SM}}$ be the BESS’s bid power in the spot market. We
define BESS’s revenue from the spot market as

$R^{\text{Bat}}=\Delta
t\sum_{t=1}^{T}\left(v_{t}^{\text{Dch}}-v_{t}^{\text{Ch}}\right)\lambda_{t}p_{t}^{\text{Bat,SM}}.$
(3)

In the co-located solar-battery system, besides power bought from the spot
market, the BESS can also store the otherwise curtailed solar generation from
the onsite solar farm. We denote the power planned to charge the BESS from
solar curtailments as $\hat{p}_{t}^{\text{Bat,SC}}$. As discussed, the BESS
cannot concurrently charge and discharge. Thus, the BESS cannot absorb
curtailed solar power while exporting its own power to the spot market. Such
an operational constraint can be formulated as

$v_{t}^{\text{Dch}}\hat{p}_{t}^{\text{Bat,SC}}=0$ (4)

The onsite curtailed solar power can be expressed as

$p_{t}^{\text{S,SC}}=\left(p_{t}^{\text{S,Act}}-p_{t}^{\text{S}}\right)\mathbb{I}\left(p_{t}^{\text{S,Act}}>p_{t}^{\text{S}}\right),$
(5)

where $\mathbb{I}(p_{t}^{\text{S,Act}}>p_{t}^{\text{S}})$ indicates the
occurrence of a solar curtailment event. Therefore, the actual power that the
BESS draws from the onsite solar farm can be presented as

$p_{t}^{\text{Bat,SC}}=\min\\{\hat{p}_{t}^{\text{Bat,SC}},p_{t}^{\text{S,SC}}\\}.$
(6)

Moreover, frequent charge and discharge of the BESS leads to battery
degradation. We introduce a non-linear battery degradation model proposed in
[19]. Let $k^{\text{deg}}_{t}$ be the coefficient associated with battery
calendar aging and cycle aging, which can be calculated using the rainflow
cycle-counting algorithm [20]. After a period of charging and discharging
operations (with a period length of $H$), the remaining battery storage
capacity (i.e., the maximum energy that can be stored in the BESS) can be
written as

$E_{{t+H},\text{max}}^{\text{Bat}}=E_{t,\text{max}}^{\text{Bat}}e^{-k^{\text{deg}}_{t}},$
(7)

where $E_{t,\text{max}}^{\text{Bat}}$ is the battery storage capacity before
arbitrage operations. With the capacity decline, we further estimate the
battery degradation coefficient, denoted by $d^{\text{deg}}_{t}$, for a given
period of arbitrage operations [21], which can be formulated as

$d^{\text{Deg}}_{t^{\prime}}=\frac{c\left(E_{t^{\prime},\text{max}}^{\text{Bat}}-E_{t^{\prime}+H,\text{max}}^{\text{Bat}}\right)}{\Delta
t\sum_{t=t^{\prime}}^{t^{\prime}+H}\left|p_{t}^{\text{Bat,SM}}+p_{t}^{\text{Bat,SC}}\right|},$
(8)

where $c$ is the battery cost per MWh [21]. We therefore formulate the battery
degradation cost as

$C^{\text{Bat}}=\Delta
t\sum_{t=1}^{T}d^{\text{Deg}}_{t}\left|p_{t}^{\text{Bat,SM}}+p_{t}^{\text{Bat,SC}}\right|.$
(9)

The degradation coefficient $d_{t}^{\text{deg}}$ is updated every $H$
intervals.

###  2.3 Bidding Formulation of the Solar-Battery System

The market participation of the co-located solar-battery system can be
formulated as an optimization problem, whose objective is to maximize the
overall profit from the spot market, written as

$\max\hskip 2.5ptR^{\text{S}}+R^{\text{Bat}}-C^{\text{Bat}},$ (10)

which includes the revenue of the solar farm, the revenue of the BESS through
energy arbitrage in the spot market, and the battery degradation cost of the
BESS.

Real-time bidding of the solar farm and the BESS is constrained as

$\displaystyle 0$ $\displaystyle\leq p_{t}^{\text{S}}\leq
p_{t}^{\text{S,Ava}},$ (11) $\displaystyle 0$ $\displaystyle\leq
p_{t}^{\text{Bat,SM}}\leq P_{\text{max}}^{\text{Bat}},$ (12) $\displaystyle 0$
$\displaystyle\leq\hat{p}_{t}^{\text{Bat,SC}}\leq
P_{\text{max}}^{\text{Bat}},$ (13) $\displaystyle 0$ $\displaystyle\leq
p_{t}^{\text{Bat,SM}}+\hat{p}_{t}^{\text{Bat,SC}}\leq
P_{\text{max}}^{\text{Bat}},$ (14) $\displaystyle 0$ $\displaystyle\leq
p_{t}^{\text{S}}+p_{t}^{\text{Bat,SM}}+\hat{p}_{t}^{\text{Bat,SC}}\leq\sigma\left(P_{\text{max}}^{\text{S}}+P_{\text{max}}^{\text{Bat}}\right),$
(15)

where $P_{\text{max}}^{\text{Bat}}$ and $P_{\text{max}}^{\text{S}}$ are the
rated power of the BESS and the solar farm, respectively, and $\sigma$ is the
transmission limit coefficient of the co-located solar-battery system, which
is set as $62.5\%$ by default. Equation (11) is the constraint for the solar
bid power. Equations (12) and (13) constrain the BESS bid power in the spot
market and the power intended to draw from the onsite solar curtailment.
Equation (14) indicates that the sum of the market bid and absorbed power from
the onsite solar farm cannot exceed the BESS’s rated power. Equation (15)
describes the export limit of the co-located system, subject to its installed
capacity.

The BESS’s capacity at each dispatch interval must be within its lower and
upper energy limits denoted by $E_{\text{min}}^{\text{Bat}}$ and
$E_{t,\text{max}}^{\text{Bat}}$, which can be formulated as

$E_{\text{min}}^{\text{Bat}}\leq e_{t-1}+\Delta e_{t}\leq
E_{t,\text{max}}^{\text{Bat}},$ (16)

where $e_{t-1}$ is BESS’s capacity after the last dispatch interval and
$\Delta e_{t}$ is the current energy change. Note that the upper energy limit
$E_{t,\text{max}}^{\text{Bat}}$ continuously decreases due to the battery
degradation. The energy change is due to the power exchange in the spot market
and curtailed solar energy absorption from the onsite solar farm, which can be
further expressed as

$\Delta e_{t}=\Delta
t\left[\left(v_{t}^{\text{Ch}}\eta^{\text{Ch}}-\frac{v_{t}^{\text{Dch}}}{\eta^{\text{Dch}}}\right)p_{t}^{\text{Bat,SM}}+v_{t}^{\text{Ch}}\eta^{\text{Ch}}p_{t}^{\text{Bat,SC}}\right],$
(17)

where $\eta^{\text{Ch}}$, $\eta^{\text{Dch}}$ are charging and discharging
efficiencies of the BESS.

The detailed system configuration of our co-located solar-battery system is
illustrated in Fig. 2.

Figure 2: The detailed system configuration.

##  3 Methodology

To solve the optimization problem formulated in Section 2.3 and better
characterize the synergy of the co-located system, we disentangle the system’s
bidding process and model it into two Markov decision processes (MDP) for the
solar farm and the BESS, respectively. We then present our AC-DRL network
structure and introduce the deep deterministic policy gradient (DDPG)
algorithm to jointly maximize the expected cumulative revenue of the solar-
battery system in the spot market. This section is organized as follows.
Section 3.1 presents MDP formulation, and Section 3.2 proposes our AC-DRL
coordinated bidding strategy.

###  3.1 MDP Modeling

The bidding strategy of the solar-battery system can be affected by multiple
factors, e.g., solar PV generation, spot price, and BESS’s capacity. To have a
better understanding of the interaction between the solar farm and the BESS,
we decouple the system’s bidding process into two MDPs for the solar farm and
the BESS, each of which consists of four elements: state space
$\mathbb{S}^{\text{S}}/\mathbb{S}^{\text{Bat}}$, action space
$\mathbb{A}^{\text{S}}/\mathbb{A}^{\text{Bat}}$, probability space
$\mathbb{P}^{\text{S}}/\mathbb{P}^{\text{Bat}}$, and reward space
$\mathbb{R}^{\text{S}}/\mathbb{R}^{\text{Bat}}$. Moreover, it is also
reasonable to disentangle the system’s bidding, as the solar farm and the BESS
may belong to different entities. Additionally, a single DRL agent has been
revealed to be inefficient in tackling multiple tasks (e.g., solar curtailment
management and bidding of the two assets) with only one reward signal [22].

State Space $\mathbb{S}$: As discussed, the external factors (e.g., the spot
price) and internal factors (e.g., the BESS’s capacity and historical solar
generation) should be considered as the state in the MDP models. Specifically,
the state of the solar farm includes the last spot price $\lambda_{t-1}$, the
last actual solar generation $p_{t-1}^{\text{S,Act}}$, the last solar power
deviation defined as
$p_{t-1}^{\text{S,Dev}}=p_{t-1}^{\text{S,Act}}-p_{t-1}^{\text{S}}$, and the
current hour index $h_{t}\in\\{0,\frac{1}{23},\frac{2}{23},\cdots,1\\}$. Since
there is no solar generation during night times, such an hour index can
partially characterize the effect of solar irradiation on solar output
profiles. For the BESS, its state consists of the last spot price, the last
solar deviation, the hour index, and the BESS’s capacity $e_{t-1}$. Moreover,
to achieve effective solar curtailment management, we also consider the number
of solar curtailment events (assuming there is no BESS for curtailment
reduction) within the latest $L$ dispatch intervals, denoted by
$f_{t-1}^{\text{SC}}$, in the BESS’s state. Moreover, we include the average
amount of solar curtailment (in MWh) in the last $L$ dispatch intervals, which
is denoted as $m^{\text{SC}}_{t-1}$. The states of the solar farm and the BESS
can be summarized as

$\displaystyle\bm{s}_{t}^{\text{S}}$
$\displaystyle=\left[\lambda_{t-1},p_{t-1}^{\text{S,Act}},p_{t-1}^{\text{S,Dev}},h_{t}\right],$
(18) $\displaystyle\bm{s}_{t}^{\text{Bat}}$
$\displaystyle=\left[\lambda_{t-1},e_{t-1},p_{t-1}^{\text{S,Dev}},f_{t-1}^{\text{SC}},m^{\text{SC}}_{t-1},h_{t}\right].$
(19)

Action Space $\mathbb{A}$: For the solar farm, its action is the bidding
$p_{t}^{\text{S}}$ based on the plant availability. The BESS’s actions include
the bid power in the spot market $p_{t}^{\text{Bat,SM}}$, the power planned to
draw from onsite solar curtailments $\hat{p}_{t}^{\text{Bat,SC}}$, and the
charge/discharge of the BESS $v_{t}^{\text{Ch}}/v_{t}^{\text{Dch}}$. To
stabilize the DRL training process, all power-related variables, e.g.,
$p_{t}^{\text{S}}$, $p_{t}^{\text{Bat,SM}}$, and
$\hat{p}_{t}^{\text{Bat,SC}}$, are normalized into the range of $[0,1]$ and
denoted as $a_{t}^{\text{S}}$, $a_{t}^{\text{Bat,SM}}$, and
$a_{t}^{\text{Bat,SC}}$, respectively. Such normalization operations make
action spaces inherently satisfy optimization constraints defined from (11) to
(13). The actions of the solar farm and the BESS are expressed as

$\bm{a}_{t}^{\text{S}}=\left[a_{t}^{\text{S}}\right],\leavevmode\nobreak\
\bm{a}_{t}^{\text{Bat}}=\left[v_{t}^{\text{Ch}},v_{t}^{\text{Dch}},a_{t}^{\text{Bat,SM}},a_{t}^{\text{Bat,SC}}\right].$
(20)

Probability Space $\mathbb{P}$: The probability space represents a probability
set of transitioning from the current state to the next state after taking an
action at the current time, which can be expressed as
$\mathbb{P}\left(\bm{s}_{t+1}|\bm{s}_{t},\bm{a}_{t}\right)$.

Reward Space $\mathbb{R}$: A reward mechanism is adopted in the MDP to assess
the effectiveness of the state transition, denoted by
$\mathbb{R}\leftarrow\mathbb{S}\times\mathbb{A}\times\mathbb{S}$. As the goal
of the DRL is to maximize the expected cumulative rewards of the MDP,
designing a suitable reward function plays a pivotal role in making better
bidding decisions to optimize the MDP.

To mitigate the uncertainty of solar PV generation and update accurate
dispatch targets for solar revenue maximization, the reward function for the
solar farm can be formulated as

$r_{t}^{\text{S}}=-\lambda_{t}\left|a_{t}^{\text{S}}-\frac{p_{t}^{\text{S,Act}}}{p_{t}^{\text{S,Ava}}}\right|.$
(21)

To perform effective energy arbitrage, we introduce two charge/discharge
indicators, denoted by $\mathbb{I}^{\text{Ch}}_{t}$ and
$\mathbb{I}^{\text{Dch}}_{t}$. The appropriate opportunities for arbitrage
operations can be indicated by
$\mathbb{I}_{t}^{\text{Ch}}=\text{sgn}(\bar{\lambda}_{t}-\lambda_{t})$ and
$\mathbb{I}_{t}^{\text{Dch}}=\text{sgn}(\lambda_{t}-\bar{\lambda}_{t})$,
respectively, where $\text{sgn}(\cdot)$ is the sign function and
$\bar{\lambda}_{t}$ is the exponential moving average of the spot price. The
moving average price $\bar{\lambda}_{t}$ is expressed as

$\bar{\lambda}_{t}=\tau\bar{\lambda}_{t-1}+\left(1-\tau\right)\lambda_{t},$
(22)

where $\tau\in(0,1)$ is a smoothing parameter. The proposed charge/discharge
indicators encourage the BESS to purchase energy at relatively lower spot
prices, i.e., $\lambda_{t}<\bar{\lambda}_{t}$, and sell energy at high prices,
i.e., $\lambda_{t}>\bar{\lambda}_{t}$. If the BESS does not follow such an
arbitrage guideline to bid in the spot market, the charge/discharge indicators
will be negative, resulting in negative rewards. Thus, the BESS’s reward
function for energy arbitrage can be formulated as

$r_{t}^{\text{Bat,SM}}=a_{t}^{\text{Bat,SM}}|\lambda_{t}-\bar{\lambda}_{t}|\left(\mathbb{I}_{t}^{\text{Ch}}v_{t}^{\text{Ch}}+\mathbb{I}_{t}^{\text{Dch}}v_{t}^{\text{Dch}}\right).$
(23)

Moreover, the BESS is incentivized to store otherwise curtailed solar power
from the onsite solar farm. The reward function for managing solar
curtailments is designed as

$r_{t}^{\text{Bat,SC}}=\beta\lambda_{t}\frac{p_{t}^{\text{Bat,SC}}}{P_{\text{max}}^{\text{Bat}}}\frac{f_{t-1}^{\text{SC}}}{L},$
(24)

where $\beta$ is the incentive factor for curtailment reduction.

Also, the battery degradation cost defined in (9) is transformed into the
below reward function, formulated as

$r_{t}^{\text{Bat,Deg}}=d^{\text{deg}}_{t}\left|a_{t}^{\text{Bat,SM}}+\frac{p_{t}^{\text{Bat,SC}}}{P_{\text{max}}^{\text{Bat}}}\right|.$
(25)

Combining reward functions for energy arbitrage, solar curtailments, and
battery degradation, the overall reward function for the BESS can be written
as

$r_{t}^{\text{Bat}}=r_{t}^{\text{Bat,SM}}+r_{t}^{\text{Bat,SC}}-r_{t}^{\text{Bat,Deg}}.$
(26)

optimizing the BESS’s revenue and mitigating battery degradation defined in
(3) and (9), respectively.

###  3.2 Optimizing MDPs via AC-DRL-empowered DDPG

####  3.2.1 DDPG Preliminaries

We use one of the most representative DRL algorithms, namely DDPG [23], to
learn an optimal bidding strategy denoted by $\pi(\bm{a}_{t}|\bm{s}_{t})$ to
maximize the expected return of the derived MDP. Note that we implement the
same DDPG structure for both the solar farm and the BESS. The objective of the
action strategy can be defined as

$J_{\pi}=\mathbb{E}_{\bm{s}_{t}\sim\mathbb{P},\bm{a}_{t}\sim\pi(\bm{s}_{t})}\left[R_{t}\right],$
(27)

where $R_{t}=\sum_{t^{\prime}=t}^{T}\gamma^{t^{\prime}-t}r_{t^{\prime}}$ is
the expected return from the $t$-th time frame with $\gamma$ denoted as the
discounted factor. Besides the action strategy $\pi$, the DDPG designs a
function, known as the critic function or the Q function, to gather the joint
state-action pair as input and assess its effectiveness. The critic function
can be formulated using the Bellman equation as

$\displaystyle Q\left(\bm{s}_{t},\bm{a}_{t}\right)$
$\displaystyle=\mathbb{E}_{\bm{s}_{t}\sim\mathbb{P},\bm{a}_{t}\sim\pi(\bm{s}_{t})}\left[R_{t}|\bm{s}_{t},\bm{a}_{t}\right],$
(28)
$\displaystyle=r_{t}+\gamma\mathbb{E}_{\bm{s}_{t+1}\sim\mathbb{P},\bm{a}_{t+1}\sim\pi}\left[Q\left(\bm{s}_{t+1},\bm{a}_{t+1}\right)\right].$

The DDPG algorithm estimates the action strategy and critic function using an
actor network $\pi_{\psi}(\bm{a}_{t},\bm{s}_{t})$ and a critic network
$Q_{\theta}(\bm{s}_{t},\bm{a}_{t})$ parameterized with $\psi$ and $\theta$,
respectively. The Adam optimizer is often utilized to update the parameters of
neural networks via gradient descent, where the actor network is updated as
$\psi\leftarrow\psi+\eta_{\psi}\nabla_{\psi}J_{\pi_{\psi}}$, where
$\eta_{\psi}$ is the learning rate of the actor network. The critic network is
updated by minimizing the residual error of the Bellman equation, where a
target critic network (parameterized by $\hat{\theta}$) is introduced to
estimate the right side of (28). The loss function of the critic network can
be written as

$L(\theta)=\mathbb{E}\left\\{\left\\{Q_{\theta}\left(\bm{s}_{t},\bm{a}_{t}\right)-\left[r_{t}+Q_{\hat{\theta}}\left(\bm{s}_{t+1},\bm{a}_{t+1}\right)\right]\right\\}^{2}\right\\}.$
(29)

Thus, the updating process of the critic network is defined as
$\theta\leftarrow\theta-\eta_{\theta}\nabla_{\theta}L(\theta)$, where
$\eta_{\theta}$ is the learning rate of the critic network. Moreover, the
target network is updated using parameters of the critic network periodically
in a moving average manner. The neural network adopted in DDPG is commonly a
multi-layer perceptron (MLP), consisting of multiple fully-connected neural
network layers (FCNNL).

####  3.2.2 AC-DRL Network Structure

In our co-located solar-battery system, the bidding decisions of both the
solar farm and the BESS are driven by multiple features as MDPs’ states
defined in (18) and (19). For instance, the time-varying spot price
incentivizes the BESS to buy low and sell high for energy arbitrage.
Meanwhile, its arbitrage behaviors may also be limited by solar curtailment
management since the BESS needs to allocate the charging space for both market
participation and onsite solar curtailments. By exploring the correlations
among features and emphasizing their relative importance during the decision-
making process, we can make more informed bidding decisions. This allows the
BESS to pay greater attention to critical features inside the input MDP state,
resulting in better bidding outcomes.

However, the typical MLP-DRL structure fails to capture such correlation and
importance information [24]. We are thus motivated to develop a novel AC-DRL
neural network structure for the DDPG algorithm, which consists of two main
components: a stacked attention mechanism and a multi-grained feature
convolution module, as depicted in Fig. 3. Moreover, the actor and critic
networks are unified to share most of neural network layers of the AC-DRL,
reducing the number of trainable parameters and subsequently accelerating the
model’s training and inference speed.

Figure 3: The AC-DRL framework. Figure 4: The inner structure of one MHCA with
two heads.

Stacked Attention: Before fed into the stacked attention, the input state
$\bm{s}_{t}$ first passes through a feature embedding layer as shown in Fig.
3. The input state is projected into a two-dimensional feature space using one
FCNNL, with each feature encoded as a specific one-dimensional embedding. The
embedding layer can be formulated as
$\hat{\bm{s}}_{t}=FCNNL(\bm{s}_{t})\in\mathbb{R}^{F\times F^{\prime}}$, where
$F$ is the number of features, i.e., the length of input state, and
$F^{\prime}$ is the length of feature embedding.

The stacked attention, which contains multiple multi-head convolutional
attention (MHCA), takes the embedded state as input, explores the correlations
between every two features, and calculates their mutual importance for bidding
decisions.

One MHCA divides the transformed feature space $F^{\prime}$ into multiple sub-
spaces $\frac{F^{\prime}}{h}$, also known as the heads of the MHCA, where $h$
is the number of heads. We illustrate the inner structure of one MHCA with two
heads in Fig. 4, where each head individually processes a low-dimensional
feature map. The embedded state $\hat{\bm{s}}_{t}$ passes through all heads
concurrently to create the query, key, and value matrices denoted by $Q$, $K$,
and $V$, which prepares for correlation computation. Such a linear projection
process can be defined as

$\displaystyle Q_{t,j}$
$\displaystyle=ReLU\left(W_{Q_{j}}\hat{\bm{s}}_{t}+b_{Q_{j}}\right)\in\mathbb{R}^{F\times\frac{F^{\prime}}{h}},$
(30) $\displaystyle K_{t,j}$
$\displaystyle=ReLU\left(W_{K_{j}}\hat{\bm{s}}_{t}+b_{K_{j}}\right)\in\mathbb{R}^{F\times\frac{F^{\prime}}{h}},$
(31) $\displaystyle V_{t,j}$
$\displaystyle=ReLU\left(W_{V_{j}}\hat{\bm{s}}_{t}+b_{V_{j}}\right)\in\mathbb{R}^{F\times\frac{F^{\prime}}{h}},$
(32)

where $j=1,\cdots,h$ is the index of MHCA head, $ReLU(x)=\max\\{0,x\\}$ is the
activation function, $W_{Q_{j}}$, $W_{K_{j}}$, and $W_{V_{j}}$ are learnable
weighted matrices, $b_{Q_{j}}$, $b_{K_{j}}$, and $b_{V_{j}}$ are learnable
bias matrices. The generated matrices are then employed by the convolutional
attention to calculate the correlation strength of every two feature
embeddings, which can be formulated as

$\bm{s}_{t,j}^{\text{CA}}=softmax\left(\frac{Q_{t,j}K_{t,j}^{T}}{\sqrt{F^{\prime}}}\right)V_{t,j}=W_{t,j}^{\text{att}}V_{t,j},$
(33)

where the result of the softmax operation, denoted by
$W_{t,j}^{\text{att}}\in\mathbb{R}^{F\times F}$, is known as the attention
weight matrix, whose element represents the mutual influences of a feature
pair. The multiplication between the attention matrix and the value matrix
aims to aggregate such correlation information to its original input state
$\hat{\bm{s}}_{t}$ and indicate the relative importance of each feature.
Unlike the original self-attention mechanism [24], we further deploy a
convolution layer, denoted by $Conv(x)$ to directly extract the aforementioned
correlation information. Concatenating the outputs of each head, we obtain the
final output of one MHCA as

$\bm{s}_{t,i}^{\text{MHCA}}=Concat\left(Conv\left(\bm{s}_{t,1}^{\text{CA}}\right),\cdots,Conv\left(\bm{s}_{t,h}^{\text{CA}}\right)\right),$
(34)

where $i=1,\cdots,N_{\text{MHCA}}$ is the index of MHCA and $N_{\text{MHCA}}$
is the number of MHCA in our stacked attention mechanism. Moreover, the
original input information will inevitably lose within the stacked attention
since the convolutional operation downsamples the input data. To keep the MHCA
remembering the full information, we also concatenate the transformed two-
dimensional input $\bm{s}_{t}$ with each MHCA’s output
$\bm{s}_{t,i}^{\text{MHCA}}$ (except for the last MHCA), which works as the
input of the following MHCA, as shown in the dotted blue line in Fig. 3.

Multi-grained Feature Convolution: We denote the output of the stacked
attention as $\bm{s}_{t,N_{\text{MHCA}}}^{\text{MHCA}}$. Taking advantage of
the “attention” information, we develop multi-grained convolution filters, as
shown in Fig. 3, to fully exploit and extract the hidden information of the
two-dimensional attention output for making better bidding decisions. Each
convolution filter is connected with a max-pooling layer to downsample and
aggregate the extracted features into one-dimensional space. The overall
feature convolution process can be formulated as

$\bm{s}_{t,k}^{\text{Conv}}=MaxPooling\left(\bm{h}_{\phi_{k}}*\bm{s}_{t,N_{\text{MHCA}}}^{\text{MHCA}}\right),$
(35)

where $*$ represents the convolution operator and $\bm{h}_{\phi_{k}}$ is a
two-dimensional convolution filter parameterized with $\phi_{k}$, where
$k=1,\cdots,N_{\text{Conv}}$ is the index of the convolution filter with
$N_{\text{Conv}}$ denoting the number of multi-grained convolution filters. As
the size of a convolution filter is determined by its parameters, a group of
filter parameters, i.e., $\\{\phi_{1},\cdots,\phi_{k}\\}$, enables our
convolution operation to gather various lower-scale feature maps and
subsequently extract multi-grained feature information from the underlying
attention output.

We then aggregate the extracted feature maps from each convolution filter and
generate our bidding decisions through one more FCNNL, which can be formulated
as

$\bm{a}_{t}=FCNNL\left(Concat\left(\left[\bm{s}_{t,1}^{\text{Conv}},\cdots,\bm{s}_{t,N_{\text{Conv}}}^{\text{Conv}}\right]\right)\right).$
(36)

For the critic network $Q_{\theta}$, the bidding decisions are integrated with
multi-grained convolution results, passing through a Feed-Forward network
(comprising two FCNNLs activated by the ReLU function) to produce a Q value,
as illustrated in Fig. 3.

Table 1: Initialized parameters. $\Delta t$ |  $5$ mins | $\alpha$ | $1.5$ | $c$ |  $1$ AU$/MWh
---|---|---|---|---|---
$P_{\text{max}}^{\text{Bat}}$ |  $10$ MW | $P_{\text{max}}^{\text{S}}$ |  $65$ MW | $\sigma$ | $0.625$
$E_{\text{min}}^{\text{Bat}}$ |  $0.5$ MWh | $E_{1,\text{max}}^{\text{Bat}}$ |  $9.5$ MWh | $\eta^{\text{Ch}},\eta^{\text{Dch}}$ | $0.95$
$H$ | $2016$ | $\tau$ | $0.9$ | $\beta$ | $6$
$L$ | $10$ | $\gamma$ | $0.99$ | $F^{\prime}$ | $64$
$h$ | $8$ | $N_{\text{MHCA}}$ | $2$ | $N_{\text{Conv}}$ | $5$

##  4 Experiments and Results

###  4.1 Experimental Setup

We use solar PV generation data in a five-minute resolution from the Rugby Run
Solar Farm located in Queensland, Australia in 2020. We also use the spot
price of the Queensland [25], which is the jurisdiction of the solar farm, to
ensure that all data are well aligned when training and testing our AC-DRL
method and the MLP-DRL benchmark. In particular, the first eleven months of
data in 2020 are used for training and the last month is for evaluation. The
storage capacity of the BESS is set to be $10$ MWh with its minimum and
maximum allowable state of charge being $5\%$ and $95\%$, respectively. The
period length $H$ of updating the battery degradation coefficient is set to be
one week. We used an Nvidia TITAN RTX graphics processing unit for algorithm
training. A Gaussian noise following the distribution $\mathcal{N}\sim(0,0.1)$
is adopted in the DDPG algorithm. For the hyperparameters of our DRL training,
the batch size is $512$. The learning rates for both actor and critic networks
are $8\text{e}^{-4}$. The replay buffer size is $1\text{e}^{5}$. Moreover, we
rename our AC-DRL strategy as AC-DDPG in this section (so as well for the MLP-
DRL method into MLP-DDPG), as we use other representative DRL algorithms for
comparisons in Section 4.2. The initialized parameters are provided in Table
1.

Table 2: The evaluation revenues (in AU$) of the DMPC, SMPC, MLP-DDPG, and our AC-DDPG methods. | Solar | BESS | Solar&BESS
---|---|---|---
DMPC | $533,302$ | $62,277$ | $595,579$
SMPC | $548,391$ | $64,156$ | $612,547$
MLP-DDPG | $586,413$ | $74,862$ | $661,275$
AC-DDPG (Ours) | $\bm{638,934}$ | $\bm{95,881}$ | $\bm{734,815}$
Figure 5: Evaluation revenue comparisons of the DMPC benchmark, the SMPC
benchmark, the MLP-DDPG benchmark, and our AC-DDPG.

###  4.2 Comparisons with MPC Benchmarks and DRL Baselines

To evaluate the effectiveness of the proposed AC-DDPG, besides the MLP-DDPG
benchmark, we develop the DMPC benchmark [7], which is a typical MPC-based
approach relying on deterministic forecasts. Specifically, we use a gated
recurrent unit (GRU) network to produce forecast results of solar PV
generation and the spot price in the next $50$ dispatch intervals, based on a
one-day length of historical input. The predictions are then fed into a mixed
integer linear programming solver (empowered by the Gurobi solver from the
PuLP library [26]) to solve the bidding problem defined from (10) to (16) for
the solar-battery system. As the DMPC benchmark fails to tackle the
uncertainties of solar generation and spot price, we also develop another MPC-
based benchmark, namely the stochastic MPC (SMPC) [8], to capture such
uncertainties via scenario generation. The number of scenarios is $120$. The
evaluation revenue comparisons of the benchmarks (DMPC, SMPC, and MLP-DDPG)
and our AC-DDPG are depicted in Fig. 5, with the associated results also
presented in Table 2. The evaluation outcomes show that the AC-DDPG
significantly outperforms all three benchmarks, with substantial revenue
boosts by $20\%$ for the solar farm, $54\%$ for the BESS, and $23\%$ for the
whole system, compared to the DMPC method, by $17\%$, $49\%$, $20\%$ compared
to the SMPC method, and by $9\%$, $28\%$, $11\%$ compared to the MLP-DDPG
method, respectively. Moreover, we also compare the performance of the adopted
DDPG algorithm with two representative off-policy algorithms – the twined
delayed deep deterministic policy gradient (TD3) [27] and the soft actor
critic (SAC) [28], which are trained with/without the attentive convolutional
(AC) mechanism. The results are shown in Fig. 6, where the DDPG algorithm
achieves higher bidding outcomes than the SAC and TD3 with/without the AC
mechanism. Such simulations also demonstrate the effectiveness of our AC
mechanism in improving the bidding performance of the co-located system from
two aspects: 1) there are considerable revenue boosts with the introduction of
the AC mechanism for all DRL algorithms, compared to the MLP-based neural
network structure; 2) the performance of these AC-DRL approaches surpass the
MPC-based benchmarks, i.e., DMPC and SMPC, by significant margins, suggesting
the consistency of AC-DRL framework in achieving higher financial returns for
the solar-battery system in electricity market participation.

Figure 6: Evaluation revenue comparisons of DDPG, SAC, and TD3 algorithms
with/without the AC mechanism.

Besides revenue comparisons, to further analyze solar curtailment management
of our AC-DDPG, we summarize key metrics related to curtailment management.
These metrics include the number of solar curtailment events, the number of
BESS responses to solar curtailment events (i.e., the number of times that the
BESS successfully absorbs the extra solar generation that will be otherwise
curtailed), the amount of absorbed curtailed solar energy by the BESS (in
MWh), the amount of curtailed solar energy of the solar farm (in MWh), as
shown in Fig. 7 and Table 3. Note that all these statistics are calculated
based on the whole evaluation process. The results in Fig. 7a indicate that
our AC-DDPG is more responsive to solar curtailments. It successfully responds
to approximately $93\%$ of solar curtailment events, which is notably higher
than the DMPC, SMPC, and MLP-DDPG benchmarks by $55\%$, $27\%$, and $19\%$,
respectively. Thus, the heightened responses to solar curtailments lead to a
substantial increase in the amount of absorbed onsite curtailed solar energy,
as shown in Fig. 7b. The “Curtailed” bar represents the amount of otherwise
curtailed solar energy after solar curtailment reduction, where all three
benchmarks curtail more solar PV generation than our AC-DDPG method.
Therefore, the curtailment management results demonstrate the significant
capability of the proposed AC-DDPG in reducing solar curtailments and
effectively harnessing onsite curtailed solar energy to enhance the economic
performance of the co-located solar-battery system.

###  4.3 The BESS Bidding Behavior Analysis

In this section, we analyze how the BESS balances the trade-off between energy
arbitrage in Section 4.3.1 and solar curtailment management in Section 4.3.2.

Table 3: Solar curtailment management comparisons of the DMPC, SMPC, MLP-DDPG, and our AC-DDPG methods. |  No. Of Curtail |  No. Of Response |  Absorbed Energy |  Curtailed Energy
---|---|---|---|---
DMPC | $5498$ | $2066$ |  $276$ MWh |  $579$ MWh
SMPC | $6256$ | $4129$ |  $413$ MWh |  $478$ MWh
MLP-DDPG | $2558$ | $1886$ |  $493$ MWh |  $316$ MWh
AC-DDPG (Ours) | $4502$ | $4192$ |  $\bm{883}$ MWh |  $\bm{285}$ MWh

(a) No. Of Curtail/Response.

(b) Curtailed/Absorbed Energy.

Figure 7: Solar curtailment management comparisons of the DMPC, SMPC, MLP-
DDPG, and our AC-DDPG methods.

####  4.3.1 BESS Energy Arbitrage

The BESS performs energy arbitrage by buying energy at low prices (i.e.,
charge) and selling stored energy at high prices (i.e., discharge) in the spot
market. The average hourly energy exchange with the spot market via
charging/discharging operations throughout the evaluation process is depicted
in Fig. 8, along with the average hourly spot price. Charged/discharged energy
at each hour index is accumulated via all $12$-interval operations in this
hour.

The results show that, while the DMPC and SMPC methods appear to deviate from
the buy-low-sell-high principle for arbitrage, both MLP-DDPG and AC-DDPG tend
to purchase more energy during the daytime when the spot price stays at a
relatively low level and subsequently sell energy during night peak hours,
such as 7 p.m., when the spot price experiences a rapid increase. The BESS’s
strategy of the buy-low-sell-high arbitrage is also corroborated by the higher
economic returns presented in Table 2. Furthermore, our AC-DDPG method enables
the BESS to conduct more frequent arbitrage during the daytime compared to the
MLP-DDPG benchmark, as depicted in Fig. 8. This increased frequency is be
attributed to AC-DDPG’s significant curtailment management capability. We
illustrate the average hourly absorbed energy (in MWh) of the BESS from solar
curtailments in Fig. 9, where our AC-DDPG enables the BESS to more effectively
take advantage of otherwise curtailed solar energy. The absorption of
curtailed solar energy effectively charges the BESS (sometimes without buying
energy from the spot market) and ensures a sufficient energy storage level for
arbitrage operations.

Figure 8: BESS’s average hourly energy exchange with spot market. Figure 9:
BESS’s average hourly absorbed solar energy.

####  4.3.2 Solar Curtailment Management

We include the number of solar curtailment events (if there is no BESS) within
the latest $10$ intervals in the BESS’s state, as defined in (19), aiming to
inform the BESS about the frequency of solar curtailments. By considering the
time-varying number of curtailment events, the BESS can dynamically allocate
the ratios between market bid $a_{t}^{\text{Bat,SM}}$ and power planned to be
drawn from onsite solar curtailment $a_{t}^{\text{Bat,SC}}$, striking a
balance between energy arbitrage and curtailment management. The relationship
between the average hourly number of curtailment events and the average
allocated charging space for absorbing curtailment $a_{t}^{\text{Bat,SC}}$
during the evaluation process is depicted in Fig. 10. The results demonstrate
that both MLP-DDPG and AC-DDPG enable the BESS to adjust the charging space
allocated for solar curtailment reduction based on the recent occurrences of
curtailments. As the number of curtailment events decreases, both methods
allocate smaller charging space, while an increase in curtailment occurrences
leads to a larger allocation. In contrast to MLP-DDPG, our AC-DDPG tends to be
more proactive, allocating more charging space to account for potential onsite
solar curtailments.

Figure 10: The solar curtailment management of MLP-DDPG and AC-DDPG under
time-varying number of curtailment events. Figure 11: The solar curtailment
management of DMPC, SMPC, MLP-DDPG, and AC-DDPG under time-varying spot
prices.

BESS’s solar curtailment management is also heavily influenced by the time-
varying spot price. The reason is twofold. Firstly, energy arbitrage is the
most direct revenue stream for the BESS and the arbitrage operation is
sensitive to the spot price changes, as illustrated in Fig. 8. Secondly, while
the BESS allocates partial charging space to absorb the onsite curtailed solar
generation instead of solely purchasing power from the spot market, solar
curtailments may not occur due to the inherent variability of solar
generation. Thus, coordinating energy arbitrage and curtailment reduction
under highly stochastic spot prices is essential for maximizing profitability.
We illustrate the impact of the average hourly spot price on curtailment
management throughout the evaluation process in Fig. 11. For the three
benchmarks, as the spot prices increase (e.g., from $3$ p.m. to $8$ p.m.), the
BESS tends to allocate less charging space for curtailment management, i.e.,
lower $a_{t}^{\text{Bat,SC}}$. In contrast, our AC-DDPG allocates more
charging space under the same conditions. This indicates that our AC-DDPG
favors curtailed solar energy during periods of higher spot price, as charging
at higher spot prices is likely to result in significant economic losses.
Using curtailed energy, though subject to availability, is free. Additionally,
the BESS allocates sufficient charging space for the onsite solar
curtailments, when the spot price is low and solar curtailments are likely to
happen, such as from $10$ a.m. to $12$ a.m.

##  5 Conclusion

This paper developed an effective coordination strategy for the co-located
solar-battery system to enhance the system’s economic performance in the spot
market. We proposed a model-free DRL-based bidding strategy, namely AC-DRL,
for the solar farm and the BESS to dynamically balance the trade-off between
energy arbitrage and solar curtailment reduction, leveraging an attention
mechanism and multi-grained feature convolution for feature correlation
extraction. We validated the proposed AC-DRL using realistic solar farm data
in the Australian NEM. Simulation results demonstrate that our method
outperforms both the MPC-based and the DRL-based benchmarks in terms of
achieving higher economic returns and improved management of solar
curtailment.

Our work also reveals several noteworthy observations: 1) The BESS
strategically purchases more power to charge itself during daytime and sells
power at night peak hours to take advantage of price spreads, adhering to the
buy-low-sell-high arbitrage principle; 2) The BESS tends to allocate more
charging space for potential solar curtailments in the presence of higher
likelihood of solar curtailment events occurring; 3) the BESS tends to use
more curtailed solar energy when the spot price rapidly increases; 4) the
time-varying spot price is the main driver for the BESS’s bidding decision-
making.

## References

  * [1] S. Comello and S. Reichelstein, “The emergence of cost effective battery storage,” Nature Communications, vol. 10, p. 2038, May 2019.
  * [2] D. S. Mallapragada, N. A. Sepulveda, and J. D. Jenkins, “Long-run system value of battery energy storage in future grids with increasing wind and solar generation,” Applied Energy, vol. 275, p. 115390, 2020.
  * [3] A. Bumpus and S. Comello, “Emerging clean energy technology investment trends,” Nature Climate Change, vol. 7, no. 6, pp. 382–385, 2017.
  * [4] J. Graça Gomes, J. Jiang, C. T. Chong, J. Telhada, X. Zhang, S. Sammarchi, S. Wang, Y. Lin, and J. Li, “Hybrid solar PV-wind-battery system bidding optimisation: A case study for the iberian and italian liberalised electricity markets,” Energy, vol. 263, p. 126043, 2023.
  * [5] A. Attarha, N. Amjady, and S. Dehghan, “Affinely adjustable robust bidding strategy for a solar plant paired with a battery storage,” IEEE Transactions on Smart Grid, vol. 10, no. 3, pp. 2629–2640, 2019.
  * [6] AEMO, 2022-23 AEMO Budget and Fees.  Australian Energy Market Operator, 2023.
  * [7] M. D. Badoual and S. J. Moura, “A learning-based optimal market bidding strategy for price-maker energy storage,” in 2021 American Control Conference (ACC), pp. 526–532, 2021.
  * [8] F. Conte, S. Massucco, M. Saviozzi, and F. Silvestro, “A stochastic optimization method for planning and real-time control of integrated PV-storage systems: Design and experimental validation,” IEEE Transactions on Sustainable Energy, vol. 9, no. 3, pp. 1188–1197, 2018.
  * [9] Y. Xie, W. Guo, Q. Wu, and K. Wang, “Robust MPC-based bidding strategy for wind storage systems in real-time energy and regulation markets,” International Journal of Electrical Power & Energy Systems, vol. 124, p. 106361, 2021.
  * [10] Y. Li, D. M. Vilathgamuwa, D. E. Quevedo, C. F. Lee, and C. Zou, “Ensemble nonlinear model predictive control for residential solar battery energy management,” IEEE Transactions on Control Systems Technology, vol. 31, no. 5, pp. 2188–2200, 2023.
  * [11] R. Weron, “Electricity price forecasting: A review of the state-of-the-art with a look into the future,” International Journal of Forecasting, vol. 30, no. 4, pp. 1030–1081, 2014.
  * [12] Y. Wang, D. Qiu, M. Sun, G. Strbac, and Z. Gao, “Secure energy management of multi-energy microgrid: A physical-informed safe reinforcement learning approach,” Applied Energy, vol. 335, p. 120759, 2023.
  * [13] A. Dolatabadi, H. Abdeltawab, and Y. A.-R. I. Mohamed, “Deep reinforcement learning-based self-scheduling strategy for a CAES-PV system using accurate sky images-based forecasting,” IEEE Transactions on Power Systems, vol. 38, no. 2, pp. 1608–1618, 2023.
  * [14] J. Li, C. Wang, and H. Wang, “Deep reinforcement learning for wind and energy storage coordination in wholesale energy and ancillary service markets,” Energy and AI, vol. 14, p. 100280, 2023.
  * [15] AEMO, How the National Electricity Market works.  Australian Energy Market Operator, 2020.
  * [16] Z. Zhu, Z. Hu, K. W. Chan, S. Bu, B. Zhou, and S. Xia, “Reinforcement learning in deregulated energy market: A comprehensive review,” Applied Energy, vol. 329, p. 120212, 2023.
  * [17] A. Daggett, M. Qadrdan, and N. Jenkins, “Feasibility of a battery storage system for a renewable energy park operating with price arbitrage,” in 2017 IEEE PES Innovative Smart Grid Technologies Conference Europe (ISGT-Europe), pp. 1–6, 2017.
  * [18] AEMO, NEM Operational Forecasting and Dispatch Handbook for wind and solar generators.  Australian Energy Market Operator, 2022.
  * [19] B. Xu, A. Oudalov, A. Ulbig, G. Andersson, and D. S. Kirschen, “Modeling of lithium-ion battery degradation for cell life assessment,” IEEE Transactions on Smart Grid, vol. 9, no. 2, pp. 1131–1140, 2018.
  * [20] M. Musallam and C. M. Johnson, “An efficient implementation of the rainflow counting algorithm for life consumption estimation,” IEEE Transactions on Reliability, vol. 61, no. 4, pp. 978–986, 2012.
  * [21] J. Cao, D. Harrold, Z. Fan, T. Morstyn, D. Healey, and K. Li, “Deep reinforcement learning-based energy storage arbitrage with accurate lithium-ion battery degradation model,” IEEE Transactions on Smart Grid, vol. 11, no. 5, pp. 4513–4521, 2020.
  * [22] F. Bai, H. Zhang, T. Tao, Z. Wu, Y. Wang, and B. Xu, “Picor: Multi-task deep reinforcement learning with policy correction,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, pp. 6728–6736, Jun. 2023.
  * [23] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” in 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016.
  * [24] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems, vol. 30, 2017.
  * [25] AEMO, “NEM data dashboard,” 2023.
  * [26] S. Mitchell, M. OSullivan, and I. Dunning, “PuLP: a linear programming toolkit for python,” The University of Auckland, Auckland, New Zealand, vol. 65, 2011.
  * [27] S. Fujimoto, H. Hoof, and D. Meger, “Addressing function approximation error in actor-critic methods,” in International Conference on Machine Learning, pp. 1582–1591, 2018.
  * [28] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” in Proceedings of the 35th International Conference on Machine Learning, vol. 80 of Proceedings of Machine Learning Research, pp. 1861–1870, PMLR, 10–15 Jul 2018.

[]Jinhao Li received a B.E. degree in smart grid information engineering with
a double B.S. degree in mathematics from the University of Electronic Science
and Technology of China (UESTC) in 2022. He is currently pursuing his Ph.D.
degree in Information Technology at the Department of Data Science and AI,
Faculty of Information Technology, Monash University. His research interest is
in machine learning for energy systems.

[]Changlong Wang is a research fellow at Monash University, specializing in
energy system modelling. His Economic Fairways Mapper team was awarded the
prestigious Australian Eureka Prize in 2023. Dr Wang is also a Climate Future
Fellow at the University of Melbourne and a visiting scholar at the University
of Oxford. He represents Australia on multiple IEA Hydrogen TCP tasks on
hydrogen modelling.

[]Hao Wang (M’16) received his Ph.D. in Information Engineering from The
Chinese University of Hong Kong in Dec. 2016. He was a Postdoctoral Research
Fellow at Stanford University and a Washington Research Foundation Innovation
Fellow at the University of Washington. He has been a Senior Lecturer at the
Department of Data Science and AI, Faculty of IT, Monash University and is now
an ARC DECRA Fellow. His research interests are in optimization, machine
learning, and data analytics for power and energy systems.