<p>Since the current network settings define one slot per second and each block that reaches a node was created for a specific slot, the local node can determine how &quot;on time&quot; the block arrived.</p>
<p>Let's first define a common understanding of the block distribution process before we get into the meaning of these new metrics:</p>
<p>Each block-generating node, when elected as a leader for a slot, has a time budget of one second to validate the accumulated transactions and scripts, generate the block, and broadcast it.</p>
<p>This is usually done via its relays, which also validate the block, adopt it, and then re-offer it to its peers for download.</p>
<p>After 2 to possibly several hops, the block will reach the block creator destined as leader for one of the upcoming slots. This can already be the next second, but on average it happens every 20 seconds according to current parameters.</p>
<p>Each node in this delivery chain will try to complete its part of the work as quickly as possible. Larger blocks with multiple transactions mean more work and therefore longer validation and transmission times. In addition, there are certain phases in each epoch where each node performs additional tasks in the background, such as the epoch switch itself and the reward calculation which currently starts 48 hours after each epoch switch and is then performed in smaller sub-steps for about 18-22 hours. This background work currently also has a noticeable influence on the processing and distribution times of the blocks, as can be seen in the following example graphics.</p>
<p>All these new metrics are derived from the first time a new block for a given slot reached the node.</p>
<p><code>_blockdelay_s</code>
is the delay between for which slot the last block received was produced and when it actually reached the local node (without the additional time required to validate it localy). This value will get updated the next time a new block for a new slot reaches the local node.</p>
<p><code>_lateblocks</code>
counting blocks that arrived more than 5s after they where supposed to have been forged. Ideally the block should arrive within the same slot = second. Larger values mean a certain risk for the block producer to not see the very last produced block in time, and build their own block based on the second last produced block, which means the following producer and whole chain may have to solve a fork. So this metric idealy should always remain at 0. A relative high growth compared to other nodes means this node has some issues to get the blocks in time.</p>
<p><code>_blockdelay_cdfOne  _blockdelay_cdfThree  _blockdelay_cdfFive</code>
These three metrics are averages based on the blocks received in the last 6 hours. So they also have a corresponding creeping progression.</p>
<p>Shown are how many blocks arrived within one, three and five seconds. Ideally, all blocks should arrive within one second = slot. You can see in the following graphics that this is largely the case, but especially in the reward calculation window it drops very clearly, because of the background activities mentioned above.
The three-second value is less affected by this. And once again significantly more blocks reach the node within 5 seconds. For the security specification of the Ouroboros Praos protocol, this 5-second value is the important and critical one. It is important that the vast majority of all blocks (95%) have reached every other block creator in the global network within 3 or max 5 seconds.</p>
<p>Some example graphs</p>
<p>The first graph shows <code>_blockdelay_cdfOne</code> for 3 different nodes, together with the processed block sizes (pink) in a timerange of 20 days = 5 epochs.
85-95% of all blocks arrive within one second.</p>
<p><a href="https://i.sstatic.net/fX8Pg.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/fX8Pg.png" alt="enter image description here" /></a></p>
<p>The drop 48h after each epoch switch because of the reward calculation load is clearly visible, lowering the receive-within-1sec-rate to 5-15%
Also visible the relation of large-block phases delaying the block propagation and lowering the 1sec-rate.</p>
<p>The following graph shows <code>_blockdelay_cdfThree</code> and <code>_blockdelay_cdfFive</code> for the same 20-day time range:</p>
<p><a href="https://i.sstatic.net/LXSyd.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/LXSyd.png" alt="enter image description here" /></a></p>
<p>We see only a slight effect on the 3-second and almost no effect on the 5-second metric.</p>
<p>The following graph zooms in to a couple of minutes, and shows the direct relation of individual block sizes (pink), and the resulting <code>_blockdelay_s</code></p>
<p><a href="https://i.sstatic.net/vhwBO.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/vhwBO.png" alt="enter image description here" /></a></p>
<p>Last but not least the new node metrics also show <code>cardano_node_metrics_forks_int</code>
We see a slowly but steadily growing number of forks. A natural and expected thing in Ouroboros proability base decentralised slot leader election. But we also see the growth rate increasing in the reward-calculation window, which means this is caused by extended propagation delays and certain blocks produced without having received the very last block in time.</p>
<p><a href="https://i.sstatic.net/EeR45.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/EeR45.png" alt="enter image description here" /></a></p>
<p><strong>Update 2022-01-11</strong></p>
<p>Since pools started upgrading to node version 1.33, the previously described 48h window for the reward calculation becomes re-scheduled to cover almost the whole epoch. This will result in less impact on the 48h window.</p>
<p>The following graphs shows this together with a second effect we currently see reducing the CDF1 ratio: more large blocks.</p>
<p>At the moment of this writing, ~ 64% of all nodes was updated to v 1.33</p>
<p><a href="https://i.sstatic.net/3TAjw.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/3TAjw.png" alt="enter image description here" /></a></p>

CC BY-SA 4.0
2022-01-03T12:01:04.337Z
Note: Starting with node version 1.33.x, the behavior in the 48-hour window will improve significantly because the reward calculation is implemented differently.
Markus - CLIO1
CC BY-SA 4.0
2022-01-10T02:28:33.493Z
A huge "thank you" for the detailed explanation.   8 hours after restarting the bp node with version 1.33.0, the numbers aren't the greatest cardano_node_metrics_blockfetchclient_blockdelay_cdfThree 0.9925925925925926
cardano_node_metrics_blockfetchclient_blockdelay_cdfOne 0.275
cardano_node_metrics_blockfetchclient_lateblocks 4
cardano_node_metrics_blockfetchclient_blockdelay_cdfFive 0.9972222222222222
cardano_node_metrics_blockfetchclient_blockdelay_s 1.75081688
Omaha Cardano Pool
CC BY-SA 4.0
2022-01-10T14:25:12.403Z
Fantastic answer.
74d4
CC BY-SA 4.0
2022-01-11T19:20:15.083Z
@OmahaCardanoPool I just updated the answer, to explain the newly introduced behaviour and effect of node version 1.33
At the timestamp of your comment, my nodes had CDF1 ratios between 0.3 and 0.6
so your 0.275 is not a real bad value for that specific moment.
Markus - CLIO1
CC BY-SA 4.0
2022-02-20T23:45:31.193Z
@Markus-CLIO1  Since the Feb 7 restart of the BP node, there have been 101 late blocks.  Reason to be concerned?   cardano_node_metrics_blockfetchclient_lateblocks 101
Omaha Cardano Pool
CC BY-SA 4.0
2022-02-22T06:47:59.880Z
This sounds like a normal value.
Markus - CLIO1
CC BY-SA 4.0
2021-12-30T09:01:40.207Z
2022-01-11T19:17:30.890Z
2022-01-11T19:17:30.890Z
Markus - CLIO1
<p>Searched docs.cardano.org, github, forum.cardano.org and the rest of the internet for some type of documentation that describes each of the items in the EKG metrics, but couldn't find anything.  I understand about half the metrics, but there are some new ones with version 1.31.0 of whose meaning I don't know. Would definitely like to know what these mean</p>
<p>cardano_node_metrics_blockfetchclient_lateblocks
cardano_node_metrics_blockfetchclient_blockdelay_s
cardano_node_metrics_blockfetchclient_blockdelay_cdfOne
cardano_node_metrics_blockfetchclient_blockdelay_cdfFive</p>

CC BY-SA 4.0
2021-12-10T19:02:25.950Z
Wow, I would really like to see that documentation!
Jay Imerman
<cardano-node>
EKG metrics interpretation and documentation
# EKG metrics interpretation and documentation

(Asked by: username_0 on 2021-11-14)

Searched docs.cardano.org, github, forum.cardano.org and the rest of the internet for some type of documentation that describes each of the items in the EKG metrics, but couldn't find anything. I understand about half the metrics, but there are some new ones with version 1.31.0 of whose meaning I don't know. Would definitely like to know what these mean

cardano\_node\_metrics\_blockfetchclient\_lateblocks
cardano\_node\_metrics\_blockfetchclient\_blockdelay\_s
cardano\_node\_metrics\_blockfetchclient\_blockdelay\_cdfOne
cardano\_node\_metrics\_blockfetchclient\_blockdelay\_cdfFive

**Tags:** cardano-node
> **Comments:**
> **username_1**: Wow, I would really like to see that documentation!  

---

## 1 answers

### Answer by username_2 on 2021-12-30. Score: 10 | Accepted answer

Since the current network settings define one slot per second and each block that reaches a node was created for a specific slot, the local node can determine how "on time" the block arrived.

Let's first define a common understanding of the block distribution process before we get into the meaning of these new metrics:

Each block-generating node, when elected as a leader for a slot, has a time budget of one second to validate the accumulated transactions and scripts, generate the block, and broadcast it.

This is usually done via its relays, which also validate the block, adopt it, and then re-offer it to its peers for download.

After 2 to possibly several hops, the block will reach the block creator destined as leader for one of the upcoming slots. This can already be the next second, but on average it happens every 20 seconds according to current parameters.

Each node in this delivery chain will try to complete its part of the work as quickly as possible. Larger blocks with multiple transactions mean more work and therefore longer validation and transmission times. In addition, there are certain phases in each epoch where each node performs additional tasks in the background, such as the epoch switch itself and the reward calculation which currently starts 48 hours after each epoch switch and is then performed in smaller sub-steps for about 18-22 hours. This background work currently also has a noticeable influence on the processing and distribution times of the blocks, as can be seen in the following example graphics.

All these new metrics are derived from the first time a new block for a given slot reached the node.

`_blockdelay_s`
is the delay between for which slot the last block received was produced and when it actually reached the local node (without the additional time required to validate it localy). This value will get updated the next time a new block for a new slot reaches the local node.

`_lateblocks`
counting blocks that arrived more than 5s after they where supposed to have been forged. Ideally the block should arrive within the same slot = second. Larger values mean a certain risk for the block producer to not see the very last produced block in time, and build their own block based on the second last produced block, which means the following producer and whole chain may have to solve a fork. So this metric idealy should always remain at 0. A relative high growth compared to other nodes means this node has some issues to get the blocks in time.

`_blockdelay_cdfOne _blockdelay_cdfThree _blockdelay_cdfFive`
These three metrics are averages based on the blocks received in the last 6 hours. So they also have a corresponding creeping progression.

Shown are how many blocks arrived within one, three and five seconds. Ideally, all blocks should arrive within one second = slot. You can see in the following graphics that this is largely the case, but especially in the reward calculation window it drops very clearly, because of the background activities mentioned above.
The three-second value is less affected by this. And once again significantly more blocks reach the node within 5 seconds. For the security specification of the Ouroboros Praos protocol, this 5-second value is the important and critical one. It is important that the vast majority of all blocks (95%) have reached every other block creator in the global network within 3 or max 5 seconds.

Some example graphs

The first graph shows `_blockdelay_cdfOne` for 3 different nodes, together with the processed block sizes (pink) in a timerange of 20 days = 5 epochs.
85-95% of all blocks arrive within one second.

[![enter image description here](https://i.sstatic.net/fX8Pg.png)](https://i.sstatic.net/fX8Pg.png)

The drop 48h after each epoch switch because of the reward calculation load is clearly visible, lowering the receive-within-1sec-rate to 5-15%
Also visible the relation of large-block phases delaying the block propagation and lowering the 1sec-rate.

The following graph shows `_blockdelay_cdfThree` and `_blockdelay_cdfFive` for the same 20-day time range:

[![enter image description here](https://i.sstatic.net/LXSyd.png)](https://i.sstatic.net/LXSyd.png)

We see only a slight effect on the 3-second and almost no effect on the 5-second metric.

The following graph zooms in to a couple of minutes, and shows the direct relation of individual block sizes (pink), and the resulting `_blockdelay_s`

[![enter image description here](https://i.sstatic.net/vhwBO.png)](https://i.sstatic.net/vhwBO.png)

Last but not least the new node metrics also show `cardano_node_metrics_forks_int`
We see a slowly but steadily growing number of forks. A natural and expected thing in Ouroboros proability base decentralised slot leader election. But we also see the growth rate increasing in the reward-calculation window, which means this is caused by extended propagation delays and certain blocks produced without having received the very last block in time.

[![enter image description here](https://i.sstatic.net/EeR45.png)](https://i.sstatic.net/EeR45.png)

**Update 2022-01-11**

Since pools started upgrading to node version 1.33, the previously described 48h window for the reward calculation becomes re-scheduled to cover almost the whole epoch. This will result in less impact on the 48h window.

The following graphs shows this together with a second effect we currently see reducing the CDF1 ratio: more large blocks.

At the moment of this writing, ~ 64% of all nodes was updated to v 1.33

[![enter image description here](https://i.sstatic.net/3TAjw.png)](https://i.sstatic.net/3TAjw.png)

> **Comments:**
> **username_2**: Note: Starting with node version 1.33.x, the behavior in the 48-hour window will improve significantly because the reward calculation is implemented differently.  
> **username_0**: A huge "thank you" for the detailed explanation.   8 hours after restarting the bp node with version 1.33.0, the numbers aren't the greatest cardano_node_metrics_blockfetchclient_blockdelay_cdfThree 0.9925925925925926 cardano_node_metrics_blockfetchclient_blockdelay_cdfOne 0.275 cardano_node_metrics_blockfetchclient_lateblocks 4 cardano_node_metrics_blockfetchclient_blockdelay_cdfFive 0.9972222222222222 cardano_node_metrics_blockfetchclient_blockdelay_s 1.75081688  
> **username_3**: Fantastic answer.  
> **username_2**: @OmahaCardanoPool I just updated the answer, to explain the newly introduced behaviour and effect of node version 1.33 At the timestamp of your comment, my nodes had CDF1 ratios between 0.3 and 0.6 so your 0.275 is not a real bad value for that specific moment.  
> **username_0**: @Markus-CLIO1  Since the Feb 7 restart of the BP node, there have been 101 late blocks.  Reason to be concerned?   cardano_node_metrics_blockfetchclient_lateblocks 101  
> **username_2**: This sounds like a normal value.  

---

