# Sustainable broadcasting in Blockchain Network with Reinforcement Learning

Danila Valko  L3S Research Center, Hannover, Germany
<EMAIL_ADDRESS>Daniel Kudenko  L3S Research Center, Hannover, Germany
<EMAIL_ADDRESS>
###### Abstract

Recent estimates put the carbon footprint of Bitcoin and Ethereum at an
average of 64 and 26 million tonnes of $CO_{2}$ per year, respectively. To
address this growing problem, several possible approaches have been proposed
in the literature: creating alternative blockchain consensus mechanisms,
applying redundancy reduction techniques, utilizing renewable energy sources,
and employing energy-efficient devices, etc. In this paper, we follow the
second avenue and propose an efficient approach based on reinforcement
learning that improves the block broadcasting scheme in blockchain networks.
Such an improvement concerns both the block propagation time and the number of
messages to be broadcast before the network reaches consistency. The latter
determines the amount of traffic and distributed energy consumption across the
blockchain network infrastructure.

In particular, we implemented a reinforcement learning (RL) agent to
efficiently re-prioritize the broadcast order in the default blockchain block
propagation protocol based on real-time transport layer network information.
This approach reduces the average propagation time as well as the number of
messages needed to reach network consistency. Since blockchain networks are
highly distributed around the world, this seemingly small improvement could
make a big difference in the context of overall energy consumption and the
network’s impact on the environment.

To train the agent and validate the approach, we used a blockchain simulator
that was extended by adding a broadcast monitoring interface and was
integrated into the RL environment. We then ran a series of simulations and
general statistical analyses to compare the performance of the default block
propagation scheme and the scheme with the RL agent involved. The analysis and
experimental results confirmed that the proposed improvement of the block
propagation scheme could cleverly handle network dynamics and achieve better
results than the default approach. Additionally, our technical integration of
the simulator and developed RL environment can be used as a complete solution
for further study of new schemes and protocols that use RL or other ML
techniques.

###### Index Terms:

Blockchain, Ethereum, Propagation protocol, Reinforcement Learning,
Sustainability

##  I Introduction

Bitcoin and Ethereum are the most carbon-intensive blockchain-based P2P
payment networks. They consume a growing amount of energy (Fig. 1) and emit an
average of 64 and 26 million tons of $CO_{2}$ per year, respectively[1, 2].
Besides Bitcoin and Ethereum, several other large blockchains produce on
average more than 750 tons of $CO_{2}$ equivalent each year[2]. Consequently,
new environmental, social, and economic challenges confront inventors,
researchers, and governments[2]. Thus, every small improvement in blockchain
protocols, schemes, and algorithms may lead to significant spatiotemporally
distributed effects. Despite this, research in this direction still does not
explicitly discuss the environmental consequences of proposed inventions and
algorithmic improvements.

(a)  (b)

Figure 1: The Energy Consumption Index of Bitcoin and Ethereum

  * •

Note. It shows yearly energy consumption of Bitcoin (a) and Ethereum (b)
infrastructure measured in terawatt-hours (TWh), data was obtained from
https://ccaf.io/cbnsi/cbeci and https://digiconomist.net/ethereum-energy-
consumption.

To address this growing problem, several possible approaches for blockchain
networks have been proposed in the literature: creating alternative consensus
mechanisms, applying redundancy reduction techniques, and utilizing renewable
energy sources and energy-efficient devices[1]. In this paper, we follow the
second avenue and propose an original and efficient approach based on
reinforcement learning (RL) that improves the block broadcasting scheme in
blockchain-based networks.

The use of RL-based approaches to solve routing problems in different types of
networks has been widely discussed in the literature (see review [3]), but
they do not directly address global peer-to-peer networks such as Bitcoin and
Ethereum (see details in §II-C). In addition, we want to emphasize the
environmental impact of the proposed solution and therefore choose and discuss
relevant metrics as a proxy for the environmental impact.

The limitations of existing blockchain routing schemes were recently addressed
in [4]. They showed that RL could enhance routing adaptability by dynamically
selecting more dependable and efficient routes, but in the context of wireless
sensor networks. We extended this concept to Ethereum-related networks,
proving that a RL agent can effectively improve broadcasting order.

As we intended to perform cost-effective, controlled, and reproducible
experimentation, our approach was tested on the blockchain simulator developed
by Faria et al.[5], which was extended by implementing the default Ethereum
broadcasting mechanism. We validated the proposed scheme improvement through a
number of simulations and demonstrated that the approach reduces the average
block propagation time while maintaining a good blockchain synchronization
rate. By implementing proposed improvements the total carbon emissions could
be reduced by approximately 0.331 gCO2eq. during each broadcasting step.
Additionally, the reduction in synchronization time should also lead to a
lower energy consumption and carbon emissions.

##  II Background and related work

###  II-A Blockchain network and block propagation

It is well known that the blockchain concept was introduced with the Bitcoin
cryptocurrency system by Satoshi Nakamoto in 2008 [6]. The blockchain can be
regarded as an immutable and decentralized database maintaining a continuously
growing list of ordered records, called blocks, that are secured from
tampering and shared among participating members. The blockchain is an
important approach for open environments [7], as it is dedicated to memorizing
data, executing transactions, performing functions, and providing trust and
secure computations [8].

Bitcoin and Ethereum are the most common blockchain cryptocurrency
architectures built on an unstructured peer-to-peer (P2P) network model. The
P2P architecture of blockchain allows all cryptocurrencies to be transferred
worldwide without the need for any middleman, intermediaries, or central
server. With the distributed P2P network, anyone who wishes to participate in
the process of verifying and validating blocks can set up a node [9].

New blocks in Bitcoin and Ethereum can only join the chain when other nodes
validate these blocks, which occurs after executing a decentralized consensus
procedure [10]. The number of blocks increases over time, and these blocks are
linked together using cryptography to form a chain. Each individual block
holds a cryptographic hash of the antecedent one.

When a node initializes, it attempts to discover a set of peers to establish
outgoing or incoming transport layer network connections that are based on the
TCP protocol. These TCP connections are used for transaction and block
propagation. Each node maintains a list of peers’ IP addresses. For instance,
according to the default protocol in the Bitcoin Core project [11], an average
node in the network initiates up to 8 outgoing connections and accepts up to
117 incoming connections. Supernodes may establish more than 8 outgoing
connections [12].

The standard block-hash propagation (BPP) scheme (protocol) in the
Bitcoin/Ethereum blockchain implies that a sending node forwards a new block
to its $N$ neighbor nodes [13]. In Bitcoin, when a certain block or
transaction arrives and is verified by a node, it notifies the neighbors using
an inventory ($inv$) message to let them know that a new block or transaction
is available and ready to send. The $inv$ message consists of the hash of the
mentioned block or transaction. When a node receives such a message for a
block or transaction that it has not already seen, it replies to the $inv$
message with a ”get data” message. Upon receiving the ”get data” message, the
node transfers the block or transaction to the sender of this message (Fig. 2,
a). In Ethereum, the sending node randomly selects $\sqrt{N}$ neighbor nodes
to forward the full block directly after verifying the block head information.
It then announces the block hash to the remaining neighbor nodes after
verifying the full block. The neighbor nodes that do not have the block will
request the block header and block body successively from this sending node to
reconstruct the full block (Fig. 2, b, c).

Figure 2: Standard block-hash propagation scheme (BPP) [14]

  * •

Note. a – default propagation scheme in Bitcoin; b, c – default propagation
scheme in Ethereum.

Message broadcasting and traffic load are the most important components that
determine energy consumption in most networks, including blockchain-related
networks. However, a significant step towards sustainability in blockchain-
related networks was taken with the invention of the proof-of-stake (PoS)
consensus algorithm [15]. The idea for PoS originated as a way to create a
less energy-consuming alternative to Bitcoin’s proof-of-work algorithm (PoW),
which requires miners to solve cryptographic puzzles to verify transactions on
the blockchain and involves substantial computations leading to high energy
consumption. Nevertheless, due to the expansion of the blockchain
infrastructure and the increasing number of transactions, further improvements
are greatly needed [1].

###  II-B Reinforcement learning for distributed networks

Reinforcement learning has shown excellent ability in solving some complex
problems, including those related to blockchain [16]. It uses rewards to
enable the algorithm to continuously optimize decision-making in the learning
process, thereby learning an optimal mapping from state to action [17]. The
rewards for performing actions are delayed; that is to say, the pros and cons
of the current action cannot be judged immediately. Only after the action has
an impact on the state, the cumulative reward obtained from the execution of
the action up to a certain moment afterward is calculated. Then, the model
accomplishes the optimization of the action by reward [18].

Basically, a task in RL is modeled as a Markov Decision Process (MDP). An MDP,
denoted as $M=(S,A,T,R,\gamma)$, where $S$ and $A$ are the state and action
space respectively. $T(s,a,s^{\prime}):S\times A\times S\rightarrow[0,1]$ is
the probability of reaching state $s^{\prime}$ from state $s$ after executing
action $a$. The reward function $R(s,a,s^{\prime}):S\times A\times
S\rightarrow\mathbb{R}$ assigns a numerical reward to a state transition from
$s$ to $s^{\prime}$ with respect to the executed action $a$. A policy
$\pi(s,a):S\times A\rightarrow[0,1]$ defines how the agents should act in the
environment through the probability distribution over all actions in every
state. The decay factor $\gamma\leq 1$ is used to define the expected
discounted return $D_{t}=\sum_{k=0}^{\infty}\gamma^{k}R_{t+k}$ and the value
function
$V(s)=E_{A_{t}\sim\pi(S_{t})}[\sum_{t=0}^{\infty}\gamma^{t}R_{t}|S_{0}=s]$.
The RL agent learns a policy that maximizes the expected discounted return,
where the optimal policy has the maximum expected discounted return [19].

The use of the RL-based approach to solve routing problems in different types
of networks has been widely discussed in the literature (see for review [3]).
For example, Boyan et al. [20] first combined the Q-learning algorithm with
packet routing to dynamically learn the routing situation and find the
shortest path in the network. Gao et al. [21] proposed a multi-agent routing
algorithm with Q-learning and backpressure, where each routing node needs only
local information about the neighbor routing nodes to solve this problem.
Mayadunna et al. [22] and Yang et al. [4] proposed RL-based malicious routing
node detection schemes for mobile ad-hoc networks and wireless sensor
networks, respectively.

###  II-C Related work and sustainability concern

Several schemes and protocols have been proposed to reduce propagation time
and related traffic load in blockchain-based networks, while also ensuring
network security: compact block propagation [12], hybrid compact block
propagation [13], bodiless block propagation [23], etc. These proposals mainly
focus on algorithms and technical improvements of the Bitcoin or Ethereum core
protocol itself. Such enhancements often necessitate significant changes in
network design and infrastructure, and in some cases, even a fork of
Bitcoin/Ethereum. Therefore, our work emphasizes the network transport layer,
which is typically invariant of the underlying top-level protocols and
requires relatively small changes in client software. By slightly modifying
the previously mentioned default propagation procedure (BPP) at the
broadcasting step with a RL-based approach, technically, such an approach can
be implemented with any other propagation scheme that usually does not depend
on the transport layer of the network.

As mentioned above, recent works consider RL implementations for blockchain-
based networks to improve their methods and algorithms. For instance, in [7],
by combining RL and blockchain for IoT networks, a decentralized communication
structure for scalable and trustworthy information allocation was developed
(for a review on RL for IoT, see [24, 25]). In [26], deep RL was used to
optimize the performance of the recently invented Prism proof-of-work
blockchain protocol [27] and enhance the number of votes without violating
security and latency performance guarantees [26]. In [28], RL was used to
auto-tune network fabric in permissioned blockchain systems and demonstrated
an ability to identify optimal network configurations.

There are also a number of papers devoted to RL solutions in the context of
improving cryptocurrency operations in Bitcoin and Ethereum: cryptocurrency
exchanges [29], trading [30, 31], portfolio management [32], and for example,
detecting specific agent behaviors [29], etc. Some new works combine
blockchain technologies and RL to improve even economic and social systems,
e.g., the healthcare system [33].

However, these works do not directly address the broadcasting problem in
global peer-to-peer networks, namely Bitcoin and Ethereum, and do not pay much
attention to the overall environmental impact.

The work most closely related to ours [4] has already stated that existing
blockchain routing schemes based on randomization or fixed-order broadcasting
struggle to cope with network dynamics and, in some cases, cannot avoid poorly
connected or even malicious nodes when routing changes are made in real-time.
In the case of wireless sensor networks, they showed that RL can improve the
self-adaptability of the routing scheme by dynamically selecting more reliable
and efficient routing channels [4]. We extended this idea to Bitcoin and
Ethereum networks and showed that a well-trained RL agent can efficiently re-
prioritize initially randomized broadcasting orders to achieve network
consistency earlier and reduce possible traffic redundancy.

##  III Methods

###  III-A The proposed block propagation protocol

As mentioned earlier, the standard blockchain propagation scheme (BPP) in the
Ethereum blockchain implies that a sending node forwards a new block to
randomly selected $\sqrt{N}$ neighboring nodes after verifying the block head
information. However, this selection may not be optimal in terms of achieving
consistency quickly. In some cases, a neighboring node may have a poor direct
connection to the sending node, relatively high latency, or even be a black
hole node [4]. In such cases, it may be beneficial to send the block to
another neighbor known to have better latency (as well as other important
metrics), and the block will eventually reach the neighboring node from a
different direction.

Given this, the proposed improvement involves employing a RL agent trained to
find the best combination of $\sqrt{N}$ neighboring nodes based on network
dynamics and current latency statistics. Thus, the RL agent takes a random
list of neighboring nodes and reorders it according to current neighborhood
observations. It is important to clarify that the proposed architecture
implies a multi-agent approach, where each client of the network has its own
agent.

###  III-B Blockchain simulator, network topology and RL-agent

Our approach is tested on the blockchain simulator developed by Faria et al.
[5]. This simulator is a discrete-event simulator flexible enough to evaluate
different blockchain implementations, such as Bitcoin and Ethereum. It follows
a stochastic simulation model, capable of representing random phenomena by
sampling from a probability distribution. The network model in the simulator
is responsible for tracking the state of each node during the simulation,
establishing connection channels between nodes, and applying network latency
to the messages being exchanged. We extended the network model by implementing
BPP broadcasting in a recursive manner and modified network monitoring to
collect appropriate statistics. Consequently, the simulator reproduces the
standard broadcasting protocol more accurately.

The network latency delay in the simulator is applied depending on the
geographic location of the destination and origin nodes. Three geographic
locations with corresponding latency distributions were provided out of the
box (Ohio, Tokyo, and Ireland), and they showed identical results for average
propagation time as in real blockchain, but with a slightly oscillating
standard deviation [5]. Based on the limited number of TCP connections of the
average node (approximately 125), we constructed a minimal, but sufficiently
large fully connected P2P network architecture of size 150 nodes. Each
geographical location consisted of 50 nodes, with half of them marked as
miners with equal hash rate (this is a prerequisite for proper simulation).

Next, we constructed an OpenAI Gym RL-environment [34] and integrated a
simulator instance so that it can be independently executed at each step of
training after a RL agent performs its action (Fig. 3). We also modified the
node model of the simulator so that it requests the RL agent to reorder the
node connections list before starting broadcasting. Thus, the active RL agent
might be involved (or not) independently during the simulation.

Figure 3: RL-agent and Blockchain simulator integration

Based on the tracked information in the simulator, we represent the dynamic
network state ($s$) from a node’s perspective as an ordered set of neighbor
latencies (corresponding to the node connections list) that change after each
communication step. For technical simplicity, we use delay estimates obtained
from the transaction propagation process, which is performed before the block
propagation phase in the simulation. We define an action function as an
ordering function over the set of connections between the node and its direct
neighbors. According to BPP, each node, by default, randomizes its list of
connections and sends full blocks only to $\sqrt{N}$ neighbor nodes, and
technically, each action ($a$) represents the corresponding order for the
node’s connection list.

To train the RL agent, we employed the Proximal Policy Optimization (PPO)
approach. PPO is a family of policy optimization methods that use multiple
epochs of stochastic gradient ascent to perform each policy update. These
methods offer the stability and reliability of trust-region methods but
demonstrate better overall performance and applicability in more general
settings[35]. There is strong evidence that PPO-based agents are capable of
solving pathfinding tasks and are adept at adjusting such heuristics[36].
Thus, we utilized the OpenAI Gym environment[34] along with the Stable-
Baselines3 PPO[37] with mostly default hyperparameters.

###  III-C Key metrics and reward function

Block propagation time is the most critical metric for blockchains as it
determines the fork rate[38, 39] and also limits the frequency of payments.
For BPP in Ethereum, the block propagation time is predominantly influenced by
the transmission time of the full block[14], which can be expressed as:

$t_{bpp}=l+size\cdot(b^{-1}+t_{proc}),$

where $t_{bpp}$ indicates block propagation time or total transmission delay;
$l$ indicates network latency; $size$ indicates block size; $t_{proc}$
indicates block processing delay; $b$ denotes bandwidth.

Hence, block propagation time should be considered the most important metric
when comparing block propagation schemes. Due to the continuous block mining
and propagation process, we opt to estimate an average block propagation time
over the blocks that were successfully propagated to at least 50% of nodes in
the network (synchronized blocks) and consider this metric as a practical
approximation of network consistency – synchronization time.

The other important metric is the probability of blockchain forks or uncle
blocks. A fork occurs whenever there are two different valid blocks at the
same block height competing to form the longest blockchain. This occurs under
normal conditions when two miners solve the proof algorithm within a short
period of time from each other. In other words, blockchain forks occur as a
result of propagation delays in the global network. A faster block mining time
would make transactions faster but lead to more frequent blockchain forks,
whereas a slower block time would decrease the number of forks but make
settlement slower. Therefore, faster block propagation time reduces the
probability that the older block wins the race.

As the probability of forks depends on the block propagation time as well as
on the frequency of block introduction (mining in Bitcoin or forging in
Ethereum), to practically compare block propagation schemes, we decide to
estimate the number of introduced blocks that were successfully propagated to
over 50% of nodes (synchronized) in the network – synchronized blocks rate.

As a proxy for the propagation complexity and the environmental impact, we
decide to count the number of messages that are broadcasted through the
network. Thus, the third of our metrics is the amount of messages that were
emitted to successfully propagate the introduced blocks to over 50% of nodes
in the network – messages per synchronized block.

Based on these metrics, we have designed a reward function for our RL-agent:

$reward=\frac{\textstyle synchronized{\>}blocks{\>}rate}{\textstyle
synchronization{\>}time}$

##  IV Results

Since we intended to test whether a well-trained RL agent can efficiently re-
prioritize broadcasting order and improve the default block propagation
protocol by doing so, we ran a series of fixed-duration Ethereum blockchain
simulations and evaluated the key metrics mentioned above. To fairly compare
BPP with the RL-agent and without it, we performed $k=1000$ simulations with a
60-second duration. This duration is minimally sufficient to introduce up to 5
blocks within a single simulation with comparatively less computation time and
overall execution time.

At each of the $k$ iterations, one simulation with and one simulation without
the RL agent was run using the same random seed and network topology. We then
recorded key metrics: synchronization time, synchronized blocks rate, and
messages per synchronized block.

Simulation results confirmed that RL-based broadcasting is able to achieve
statistically significant improvements of network dynamics (Fig. 4). On
average over 1000 simulations, synchronization time was reduced by 1.7%, and
synchronized blocks rate was increased by 3.4 percentage points. The total
number of messages per synchronized block was reduced by 5.0%.

Figure 4: Main simulation results

  * •

Note. The results of 1000 simulations performed for the default block-hash
propagation (BPP) and RL-agent-based block broadcasting (RLA) are depicted in
the figures. These figures represent the cumulative fraction of empirical
values (ECDF): the higher and farther left the line, the lower the average
value. The difference is statistically significant ($p<.001$) using an
unpaired Wilcoxon rank sum test with continuity correction.

To estimate the environmental impact of proposed improvements, it is necessary
to consider various emission factors relevant to the average Ethereum block
and the transmission of related messages via the TCP protocol and Internet
infrastructure.

Firstly, there is a direct correlation between the quality of service (QoS) in
the TCP communication network and its carbon emissions. In particular, carbon
emissions escalate as QoS deteriorates[40]. This increase is attributed to the
growing number of packets generated and their subsequent transmissions via an
unstable network. An examination of a modestly sized network, consisting of
100 nodes operating under varied traffic loads and maintaining a minimum of
80% QoS, reveals an average emission rate of 0.156 gCO2eq. per byte[41]. This
figure is a yearly average based on natural gas as the primary electricity
source.

However, deriving a more precise estimate is challenging due to various
hardware in use[41]. Reference data indicates that the peak energy consumption
of modern Ethernet switches, which primarily function to route packets, stands
at approximately $4.42E{-09}$ gCO2eq. per byte[42, 43].

Secondly, the size of the average Ethereum block has shown a consistent upward
trend. Data from February 2024 indicates that the daily average block size has
reached 202,664 bytes, a significant increase from 106,063 bytes a year prior
and 25,117 bytes five years earlier[44]. Taking an average block size of
154,363 bytes and assuming that broadcasting one block equates to transmitting
one message — without considering additional technical communications and
seamless infrastructure nodes of the global Internet — it results in emissions
of $6.82E{-04}$ gCO2eq. per message per node. Considering Ethereum’s protocol,
which introduces a new block every 12-15 seconds to be broadcasted across
billions of nodes globally, the potential for emission reduction becomes
evident.

By implementing proposed improvements that achieve as much as a 5% reduction
in message transmission within our network model of 150 directly connected
nodes, the total carbon emissions could be reduced by approximately
$6.82E{-04}\times 485.125=0.331$ gCO2eq. during each broadcasting phase.
Additionally, the reduction in synchronization time should also lead to a
reduction in energy consumption and carbon emissions.

##  V Limitations and future work

Using blockchain simulations in experiments is a trade-off between realism and
control. Simulations provide a cost-effective, controlled, and safe
environment for research and experimentation, but they come with inherent
limitations in replicating the complexity of real-world blockchain networks.
For instance, simulations can never fully replicate the complexity and
dynamics of a real blockchain network; they often rely on simplifications and
assumptions, such as uniform network conditions or idealized node behaviors;
simulating large-scale blockchain networks with thousands or millions of nodes
can be computationally intensive and may not be feasible due to hardware and
software constraints etc.

Since we wanted to make not only a controllable and reproducible but also a
fair comparison between the default and our improved block propagation
protocol, we constructed a simulation-based experimental design. However, in
future work we plan to build a small real blockchain testbed, and also
consider deeply exploring the proposed approach implementation for other
Ethereum-like networks. We also plan to continue experimenting with increasing
the degrees of freedom of the solution. For example, by extending the agent’s
actions with the ability to determine the number of nodes to broadcast the
block, instead of being limited by the Ethereum protocol.

##  VI Conclusion

In this paper, we have attempted to address the environmental issues
associated with blockchain-based P2P payment networks and explicitly discuss
the environmental implications of the proposed improvements. We believe that
every small improvement in blockchain protocols and its key metrics should
also be considered in terms of energy consumption, data flow, and network
load. Therefore, in this work, we:

  1. 1.

Developed an improved Ethereum blockchain peer-to-peer network propagation
scheme that involves a RL agent to efficiently re-prioritize the broadcast
order based on real-time transport layer network information.

  2. 2.

Enhanced an event-driven blockchain simulator by involving a RL agent and
created a simulator-based integrated RL learning environment.

  3. 3.

Validated the proposed scheme improvement on a number of simulations and
showed that the approach reduces the average block propagation time while
maintaining a good blockchain synchronization rate.

  4. 4.

Emphasized the environmental impact and estimated the required number of
broadcast messages before the blockchain network achieved its consistency.

## Code and Data availability

All data and algorithms used can be found in the public repository:
https://github.com/ellariel/eth-broadcast-protocol.

## References

  * [1] V. Kohli, S. Chakravarty, V. Chamola, K. S. Sangwan, and S. Zeadally, “An analysis of energy consumption and carbon footprints of cryptocurrencies and possible solutions,” _Digital Communications and Networks_ , vol. 9, no. 1, pp. 79–89, 2023.
  * [2] “Guidelines for improving Blockchain’s environmental, social and economic impact,” World Economic Forum, 2023, retrieved June 16, 2023 from https://www3.weforum.org/docs/WEF_Guidelines_for_Improving_Blockchain%E2%80%99s_Environmental_Social_and_Economic_Impact_2023.pdf.
  * [3] Z. Mammeri, “Reinforcement learning based routing in networks: Review and classification of approaches,” _IEEE Access_ , vol. 7, pp. 55 916–55 950, 2019.
  * [4] J. Yang, S. He, Y. Xu, L. Chen, and J. Ren, “A trusted routing scheme using blockchain and reinforcement learning for wireless sensor networks,” _Sensors_ , vol. 19, no. 4, 2019.
  * [5] C. Faria and M. Correia, “Blocksim: Blockchain simulator,” in _2019 IEEE International Conference on Blockchain (Blockchain)_ , 2019, pp. 439–446.
  * [6] S. Nakamoto, “Bitcoin: A peer-to-peer electronic cash system,” 2009, retrieved February 23, 2024 from https://www.bibsonomy.org/bibtex/2974d35fdb27dea57296ed2245556aa18/daniel_grm9. [Online]. Available: http://www.bitcoin.org/bitcoin.pdf
  * [7] T. Alam, “Blockchain-enabled deep reinforcement learning approach for performance optimization on the internet of things,” _Wireless Personal Communications_ , vol. 126, pp. 995–1011, 2022.
  * [8] R. Zhang, R. Xue, and L. Liu, “Security and privacy on blockchain,” _ACM Comput. Surv._ , vol. 52, no. 3, jul 2019. [Online]. Available: https://doi.org/10.1145/3316481
  * [9] T. K. Sharma, “Blockchain & role of p2p network,” Blockchain Council, 2022, retrieved September 15, 2023 from https://www.blockchain-council.org/blockchain/blockchain-role-of-p2p-network/.
  * [10] C. Lin, D. He, N. Kumar, X. Huang, P. Vijayakumar, and K.-K. R. Choo, “Homechain: A blockchain-based secure mutual authentication system for smart homes,” _IEEE Internet of Things Journal_ , vol. 7, no. 2, pp. 818–829, 2020.
  * [11] “Bitcoin core,” Bitcoin Core Project, 2020, retrieved September 15, 2023 from https://bitcoin.org/en/bitcoin-core/.
  * [12] L. Zhang, T. Wang, and S. C. Liew, “Speeding up block propagation in blockchain network: Uncoded and coded designs,” arXiv, 2021.
  * [13] C. Zhao, T. Wang, S. Zhang, and S. C. Liew, “Hcb: Enabling compact block in ethereum network with secondary pool and transaction prediction,” arXiv, 2022.
  * [14] X. Ma, H. Wu, D. Xu, and K. Wolter, “Cblocksim: A modular high-performance blockchain simulator,” in _2022 IEEE International Conference on Blockchain and Cryptocurrency (ICBC)_ , 2022, pp. 1–5.
  * [15] S. King and S. Nadal, “Ppcoin: Peer-to-peer crypto-currency with proof-of-stake,” 2012. [Online]. Available: https://api.semanticscholar.org/CorpusID:42319203
  * [16] X. Bai, S. Tu, M. Waqas, A. Wu, Y. Zhang, and Y. Yang, “Blockchain enable iot using deep reinforcement learning: A novel architecture to ensure security of data sharing and storage,” in _Artificial Intelligence and Security_ , X. Sun, X. Zhang, Z. Xia, and E. Bertino, Eds. Cham: Springer International Publishing, 2022, pp. 586–597.
  * [17] R. Nian, J. Liu, and B. Huang, “A review on reinforcement learning: Introduction and applications in industrial process control,” _Computers & Chemical Engineering_, vol. 139, p. 106886, 2020.
  * [18] M. Hatem and F. Abdessemed, “Simulation of the navigation of a mobile robot by the qlearning using artificial neuron networks,” in _Conférence Internationale sur l’Informatique et ses Applications_ , 2009. [Online]. Available: https://api.semanticscholar.org/CorpusID:2909520
  * [19] R. S. Sutton and A. G. Barto, _Reinforcement Learning: An Introduction_ , 2nd ed. The MIT Press, 2018. [Online]. Available: http://incompleteideas.net/book/the-book-2nd.html
  * [20] J. A. Boyan and M. L. Littman, “Packet routing in dynamically changing networks: A reinforcement learning approach,” in _NIPS_ , 1993. [Online]. Available: https://api.semanticscholar.org/CorpusID:364332
  * [21] J. Gao, Y. Shen, M. Ito, and N. Shiratori, “Multi-agent q-learning aided backpressure routing algorithm for delay reduction,” arXiv, 2017.
  * [22] H. Mayadunna, S. L. D. Silva, I. Wedage, S. Pabasara, L. Rupasinghe, C. Liyanapathirana, K. K. Kesavan, C. P. Nawarathna, and K. K. Sampath, “Improving trusted routing by identifying malicious nodes in a manet using reinforcement learning,” _2017 Seventeenth International Conference on Advances in ICT for Emerging Regions (ICTer)_ , pp. 1–8, 2017. [Online]. Available: https://api.semanticscholar.org/CorpusID:40517242
  * [23] C. Zhao, S. Zhang, T. Wang, and S. C. Liew, “Bodyless block propagation: Tps fully scalable blockchain with pre-validation,” arXiv, 2022.
  * [24] R. Gasmi, S. Hammoudi, M. Lamri, and S. Harous, “Recent reinforcement learning and blockchain based security solutions for internet of things: Survey,” _Wireless Personal Communications_ , vol. 132, pp. 1307–1345, 2023.
  * [25] L. Gao, X. Zhang, T. Liu, H. Yang, B. Liao, and J. Guo, “Energy-aware blockchain resource allocation algorithm with deep reinforcement learning for trusted authentication,” in _Smart Grid and Innovative Frontiers in Telecommunications_ , M. Cheng, P. Yu, Y. Hong, and H. Jia, Eds. Cham: Springer International Publishing, 2021, pp. 93–103.
  * [26] D. S. Gadiraju, V. Lalitha, and V. Aggarwal, “An optimization framework based on deep reinforcement learning approaches for prism blockchain,” _IEEE Transactions on Services Computing_ , vol. 16, no. 4, pp. 2451–2461, 2023.
  * [27] V. Bagaria, S. Kannan, D. Tse, G. Fanti, and P. Viswanath, “Prism: Deconstructing the blockchain to approach physical limits,” in _Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security_ , ser. CCS ’19. New York, NY, USA: Association for Computing Machinery, 2019, p. 585–602.
  * [28] M. Li, Y. Wang, S. Ma, C. Liu, D. Huo, Y. Wang, and Z. Xu, “Auto-tuning with reinforcement learning for permissioned blockchain systems,” _Proc. VLDB Endow._ , vol. 16, no. 5, p. 1000–1012, jan 2023.
  * [29] M. Schnaubelt, “Deep reinforcement learning for the optimal placement of cryptocurrency limit orders,” _European Journal of Operational Research_ , vol. 296, no. 3, pp. 993–1006, 2022.
  * [30] B. J. D. Gort, X.-Y. Liu, X. Sun, J. Gao, S. Chen, and C. D. Wang, “Deep reinforcement learning for cryptocurrency trading: Practical approach to address backtest overfitting,” arXiv, 2022.
  * [31] M. Asgari and S. H. Khasteh, “Profitable strategy design by using deep reinforcement learning for trades on cryptocurrency markets,” arXiv, 2022.
  * [32] Z. Huang and F. Tanaka, “A scalable reinforcement learning-based system using on-chain data for cryptocurrency portfolio management,” arXiv, 2023.
  * [33] A. Lakhan, M. A. Mohammed, J. Nedoma, R. Martinek, P. Tiwari, and N. Kumar, “Drlbts: deep reinforcement learning-aware blockchain-based healthcare system,” _Scientific Reports_ , vol. 13, p. 4124, 2023.
  * [34] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, “Openai gym,” arXiv, 2016.
  * [35] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv, 2017.
  * [36] A. Skrynnik, A. Andreychuk, K. Yakovlev, and A. Panov, “Pathfinding in stochastic environments: learning vs planning,” _PeerJ Computer Science_ , vol. 8, p. e1056, 2022.
  * [37] “The proximal policy optimization algorithm,” Stable Baselines3 Library, 2021, retrieved September 15, 2023 from https://stable-baselines3.readthedocs.io/en/master/modules/ppo.html.
  * [38] A. Gervais, G. O. Karame, K. Wüst, V. Glykantzis, H. Ritzdorf, and S. Capkun, “On the security and performance of proof of work blockchains,” in _Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security_ , ser. CCS ’16. New York, NY, USA: Association for Computing Machinery, 2016, p. 3–16.
  * [39] U. Klarman, S. S. Basu, A. Kuzmanovic, and E. G. Sirer, “bloxroute: A scalable trustless blockchain distribution network whitepaper,” 2018. [Online]. Available: https://api.semanticscholar.org/CorpusID:92979777
  * [40] S. J. Habib and P. N. Marimuthu, “Comparing communication protocols within an enterprise network for carbon footprint reduction,” _Netw. Protoc. Algorithms_ , vol. 5, pp. 111–126, 2013. [Online]. Available: https://api.semanticscholar.org/CorpusID:37003305
  * [41] M. Usman, D. Kliazovich, F. Granelli, P. Bouvry, and P. Castoldi, “Energy efficiency of tcp: An analytical model and its application to reduce energy consumption of the most diffused transport protocol,” _International Journal of Communication Systems_ , vol. 30, no. 1, p. e2934, 2017.
  * [42] F. He, “Exploration of distributed image compression and transmission algorithms for wireless sensor networks,” _International Journal of Online and Biomedical Engineering (iJOE)_ , vol. 15, no. 01, p. 143–155, Jan. 2019.
  * [43] A. Vishwanath, F. Jalali, K. Hinton, T. Alpcan, R. W. A. Ayre, and R. S. Tucker, “Energy consumption comparison of interactive cloud-based and local applications,” _IEEE Journal on Selected Areas in Communications_ , vol. 33, no. 4, pp. 616–626, 2015.
  * [44] “Ethereum average block size,” YCharts, 2024, retrieved February 23, 2024 from https://ycharts.com/indicators/ethereum_average_block_size.