add block arrivals report base + methodology (#118)

* add block arrivals report base + methodology

* Apply suggestions from code review

Co-authored-by: Yiannis Psaras <[email protected]>

* move report to week 29 + add report formating

---------

Co-authored-by: Yiannis Psaras <[email protected]>

content.en/ethereum/block_arrival/2024-29-report.md

@@ -0,0 +1,47 @@
+---
+title: Week 2024-29
+plotly: true
+weight: 1048566
+slug: 2024-29
+aliases:
+    - /block-arrival-times/2024-09/
+---
+
+# Ethereum block broadcasting latency report 2024-29 
+
+The following results show measurement data that were collected in **calendar week** 29 **of** 2024 from `2024-06-14` to `2024-06-16`.
+
+This report provides charts and metrics for the beacon block broadcasting latency on the Ethereum network. The methodology we used is available [here](../methodology).    
+
+{{< hint info >}}
+💡 A more detailed report is available on [this ethresear.ch](https://ethresear.ch/t/gossipsub-message-propagation-latency/19982) post.
+{{< /hint >}}
+
+## Block arrival time within the slot 
+The line plot displays the Cumulative Distribution Function (CDF) of the block arrival times within the slot for the different Ethereum clients (listed [here](../methodology#data-source)). The line plot aggregates all the locations we collect data from for each client.
+
+{{< plotly json="../../../plots/latest/ethereum-block-broadcasting-latency/gossipsub_arrival_times_within_slot_by_agent_on_mainnet_beacon_block.json" height="600px" >}}
+
+The following line plot shows the CDF of the block arrival time within the slot, aggregating all the data points by the continent that the clients were in.
+
+{{< plotly json="../../../plots/latest/ethereum-block-broadcasting-latency/gossipsub_arrival_times_within_slot_on_by_continent_mainnet_beacon_block.json" height="600px" >}}
+
+## Block arrival time to block size correlation
+The following heatmap shows the concentration of block arrival times, aggregating the block size in 50KB range increments. The number displayed within each square represents the corresponding percentage of blocks falling into: i) the block size, and ii) arrival time, out of the total number of blocks seen.
+
+{{< plotly json="../../../plots/latest/ethereum-block-broadcasting-latency/message_arrival_time_and_size_correlation_heatmap_on_mainnet_beacon_block.json" height="600px" >}}
+
+## Block arrival time over epochs
+The line plot shows the maximum, minimum, mean and median values of the block arrival times aggregated over a range of 4 epochs.
+
+_NOTE: The time aggregation is done over windows of 4 epochs (or 1536 seconds)._
+
+{{< plotly json="../../../plots/latest/ethereum-block-broadcasting-latency/message_arrivals_max_min_on_1536s_window_on_topic_mainnet_beacon_block.json" height="600px" >}}
+
+The following line plot shows the time distribution of the mean block arrival time aggregated by client type and over time windows of 4 epochs.
+
+{{< plotly json="../../../plots/latest/ethereum-block-broadcasting-latency/message_arrival_time_on_1536s_window_by_agents_on_topic_mainnet_beacon_block.json" height="600px" >}}
+
+Lastly, the following line plot displays the time distribution of the mean block arrival times, aggregated by continent and over time windows of 4 epochs.
+
+{{< plotly json="../../../plots/latest/ethereum-block-broadcasting-latency/message_arrival_time_on_1536s_window_by_continent_on_topic_mainnet_beacon_block.json" height="600px" >}}

content.en/ethereum/block_arrival/_index.md

@@ -0,0 +1,4 @@
+---
+title: Block Arrival Times
+bookCollapseSection: true
+---

content.en/ethereum/block_arrival/methodology.md

@@ -0,0 +1,57 @@
+---
+title: Methodology
+weight: 10
+slug: methodology
+---
+
+# Methodology for block arrivals reports 
+
+## Basics on Ethereum time deadlines
+The Ethereum specification defines strict time deadlines on its consensus layer:
+- The spec first defines epochs, which are 6.4 minutes long and are composed of 32 slots of 12 seconds each.
+- Although not explicitly defined in the specification, each epoch is further split into three periods:
+    - Block composition and its propagation (0 - 4th second, red boxes in the plot below).
+    - Attestation composition and its propagation (4th - 8th second, blue boxes in the plot below).
+    - Aggregation of attestations and their propagation (8th - 12th second, yellow boxes in the plot below).
+
+![img](../slot_time_division.png)
+
+In theory, nodes compose the block at the slot's starting time, so the propagation starts at`t0`of the slot, giving Gossipsub 4 seconds to propagate over the block over the entire network.
+After those 4 seconds, validators must attest to their perception of the chain, including the last block they saw. Ensuring that blocks arrive before the 4-second deadline is crucial to reduce the chances of splitting the network (fork) between nodes that saw a message and nodes that didn't.
+
+
+## Why it's important to monitor block propagation latency
+The defined time windows are not hardcoded anywhere; they are just guidelines to ensure the best performance. However, it's not always in the best interest of the validators to follow them. In fact, it's often the case that validators benefit from delaying message propagation to get extra rewards from the network.
+
+The most well-known case of validators benefiting from artificially delaying the propagation of messages is the "Miner Extractable Value" or [MEV](https://coinmarketcap.com/academy/glossary/miner-extractable-value-mev). 
+The strict 12-second window that PoS defines enables validators to focus primarily on producing the most profitable blocks, by including the highest value payloads in the blocks they propose. 
+This is done by ordering the available transactions in the most profitable way. In this context, the public bids submitted to [public relays](https://ethstaker.cc/mev-relay-list) can then be selected by validators that have to propose new blocks. 
+
+## How does MEV affect the block proposal and its propagation?
+
+The more time these block builders have, the more time they have to develop a more profitable execution payload. This ultimately incentive validators to "wait" a little bit more beyond the defined `t0` mark, as the MEV reward might be substantially higher due to those extra 1-2 seconds (check out the [live dashboard](https://payload.de/data/) of MEV bids and their reward). 
+
+The most significant limitation in those cases is still the 4-second line, where validators must attest to the last block they saw. Thus, the block proposer can wait longer for a higher rewarding bid without being too greedy and exceed the point where the block propagation couldn't be done by 2/3s of the validators.
+
+## Data source
+All the data displayed in the Block Arrival Time reports was collected from a [public database](https://github.com/ethpandaops/xatu-data) maintained by the [EthPandaOps](https://ethpandaops.io/) team of the Ethereum Foundation.
+
+The raw data includes data points from a set of "sentry nodes" located in the following locations:
+
+| Continent | Country   |	Client      |
+| ---       | ---       | ---           |
+| EU        | FI        | lighthouse    |
+|           |           | lodestar      |
+|           |           | nimbus        |
+|           |           | prysm         |
+|           |           | teku          |
+| NA        | US        | lighthouse    |
+|           |           | lodestar      |
+|           |           | nimbus        |
+|           |           | prysm         |
+|           |           | teku          |
+| OC        | AU        | lighthouse    | 
+|           |           | lodestar      |
+|           |           | nimbus        |
+|           |           | prysm         |
+|           |           | teku          |  

content.en/plots/latest/ethereum-block-broadcasting-latency/message_arrival_time_and_size_correlation_heatmap_on_mainnet_beacon_block.json

@@ -0,0 +1 @@
+{"data": [{"coloraxis": "coloraxis", "name": "0", "texttemplate": "%{z}", "x": ["0.0-0.5", "0.5-1.0", "1.0-1.5", "1.5-2.0", "2.0-2.5", "2.5-3.0", "3.0-3.5", "3.5-4.0", "4.0-4.5", "4.5-5.0", "5.0-5.5", "5.5-6.0", "6.0-6.5", "6.5-7.0", "7.0-7.5", "7.5-8.0", "8.0-8.5", "8.5-9.0", "9.0-9.5", "10.0-10.5", "10.5-11.0", "11.0-11.5"], "y": ["400.0-450.0", "350.0-400.0", "300.0-350.0", "250.0-300.0", "200.0-250.0", "150.0-200.0", "100.0-150.0", "50.0-100.0", "0-50", "0.0-50.0"], "z": [[null, null, 0.013895321908290875, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], [null, null, 0.004631773969430292, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], [null, null, 0.018527095877721167, 0.018527095877721167, 0.02779064381658175, 0.004631773969430292, 0.004631773969430292, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], [null, 0.009263547938860583, 0.09726725335803613, 0.09726725335803613, 0.06484483557202408, 0.009263547938860583, null, 0.004631773969430292, 0.004631773969430292, null, 0.009263547938860583, null, 0.004631773969430292, null, null, null, null, null, null, null, null, null], [0.004631773969430292, 0.04631773969430292, 0.333487725798981, 0.41222788327929594, 0.15284854099119963, 0.04631773969430292, 0.03242241778601204, 0.004631773969430292, null, null, null, 0.004631773969430292, 0.004631773969430292, null, null, null, null, null, null, null, null, null], [0.004631773969430292, 0.037054191755442334, 0.2871699861046781, 0.3798054654932839, 0.19453450671607225, 0.06484483557202408, 0.02315886984715146, 0.009263547938860583, 0.013895321908290875, 0.004631773969430292, null, null, 0.009263547938860583, null, null, null, null, null, null, null, null, null], [null, 0.10189902732746642, 0.9541454377026402, 1.4219546086150996, 0.7966651227420102, 0.2686428902269569, 0.12042612320518759, 0.041685965724872626, 0.009263547938860583, 0.004631773969430292, 0.013895321908290875, 0.004631773969430292, null, 0.009263547938860583, 0.004631773969430292, null, null, null, null, null, null, null], [0.041685965724872626, 1.0097267253358035, 7.670217693376563, 9.249652616952293, 4.817044928207503, 1.352477999073645, 0.8105604446503011, 0.30106530801296894, 0.09726725335803613, 0.07874015748031496, 0.09726725335803613, 0.12505789717461788, 0.0555812876331635, 0.018527095877721167, 0.004631773969430292, null, null, null, null, 0.004631773969430292, null, null], [0.37054191755442334, 0.9170912459471978, 1.0884668828161186, 0.6762389995368225, 0.30106530801296894, 0.12042612320518759, 0.07410838351088467, 0.02779064381658175, 0.02315886984715146, 0.03242241778601204, 0.009263547938860583, 0.013895321908290875, 0.004631773969430292, 0.018527095877721167, 0.009263547938860583, null, null, 0.009263547938860583, 0.004631773969430292, null, 0.004631773969430292, null], [0.5697081982399259, 4.562297359888837, 22.86243631310792, 22.297359888837427, 9.444187123668366, 2.209356183418249, 1.0189902732746643, 0.3473830477072719, 0.13895321908290875, 0.3427512737378416, 0.20379805465493284, 0.17600741083835109, 0.09263547938860583, 0.037054191755442334, 0.004631773969430292, 0.013895321908290875, 0.004631773969430292, 0.004631773969430292, 0.009263547938860583, null, null, 0.004631773969430292]], "type": "heatmap", "xaxis": "x", "yaxis": "y", "hovertemplate": "arrival_t: %{x}<br>size_range: %{y}<br>color: %{z}<extra></extra>"}], "layout": {"template": {"data": {"scatter": [{"type": "scatter"}]}}, "xaxis": {"anchor": "y", "domain": [0.0, 1.0], "title": {"text": "Arrival time within slot"}}, "yaxis": {"anchor": "x", "domain": [0.0, 1.0], "autorange": "reversed", "title": {"text": "Size (KB)"}}, "coloraxis": {"colorscale": [[0.0, "#440154"], [0.1111111111111111, "#482878"], [0.2222222222222222, "#3e4989"], [0.3333333333333333, "#31688e"], [0.4444444444444444, "#26828e"], [0.5555555555555556, "#1f9e89"], [0.6666666666666666, "#35b779"], [0.7777777777777778, "#6ece58"], [0.8888888888888888, "#b5de2b"], [1.0, "#fde725"]]}, "margin": {"t": 60}, "title": {}}}

content.en/plots/latest/ethereum-block-broadcasting-latency/message_arrival_time_on_1536s_window.json

@@ -0,0 +1 @@
+{"data": [{"hovertemplate": "topic=mainnet_beacon_block<br>timestamp=%{x}<br>time_in_slot=%{y}<extra></extra>", "legendgroup": "mainnet_beacon_block", "line": {"color": "#1F77B4", "dash": "solid"}, "marker": {"symbol": "circle"}, "mode": "lines", "name": "mainnet_beacon_block", "orientation": "v", "showlegend": true, "x": ["2024-06-14T00:00:00", "2024-06-14T00:25:36", "2024-06-14T00:51:12", "2024-06-14T01:16:48", "2024-06-14T01:42:24", "2024-06-14T02:08:00", "2024-06-14T02:33:36", "2024-06-14T02:59:12", "2024-06-14T03:24:48", "2024-06-14T03:50:24", "2024-06-14T04:16:00", "2024-06-14T04:41:36", "2024-06-14T05:07:12", "2024-06-14T05:32:48", "2024-06-14T05:58:24", "2024-06-14T06:24:00", "2024-06-14T06:49:36", "2024-06-14T07:15:12", "2024-06-14T07:40:48", "2024-06-14T08:06:24", "2024-06-14T08:32:00", "2024-06-14T08:57:36", "2024-06-14T09:23:12", "2024-06-14T09:48:48", "2024-06-14T10:14:24", "2024-06-14T10:40:00", "2024-06-14T11:05:36", "2024-06-14T11:31:12", "2024-06-14T11:56:48", "2024-06-14T12:22:24", "2024-06-14T12:48:00", "2024-06-14T13:13:36", "2024-06-14T13:39:12", "2024-06-14T14:04:48", "2024-06-14T14:30:24", "2024-06-14T14:56:00", "2024-06-14T15:21:36", "2024-06-14T15:47:12", "2024-06-14T16:12:48", "2024-06-14T16:38:24", "2024-06-14T17:04:00", "2024-06-14T17:29:36", "2024-06-14T17:55:12", "2024-06-14T18:20:48", "2024-06-14T18:46:24", "2024-06-14T19:12:00", "2024-06-14T19:37:36", "2024-06-14T20:03:12", "2024-06-14T20:28:48", "2024-06-14T20:54:24", "2024-06-14T21:20:00", "2024-06-14T21:45:36", "2024-06-14T22:11:12", "2024-06-14T22:36:48", "2024-06-14T23:02:24", "2024-06-14T23:28:00", "2024-06-14T23:53:36", "2024-06-15T00:19:12", "2024-06-15T00:44:48", "2024-06-15T01:10:24", "2024-06-15T01:36:00", "2024-06-15T02:01:36", "2024-06-15T02:27:12", "2024-06-15T02:52:48", "2024-06-15T03:18:24", "2024-06-15T03:44:00", "2024-06-15T04:09:36", "2024-06-15T04:35:12", "2024-06-15T05:00:48", "2024-06-15T05:26:24", "2024-06-15T05:52:00", "2024-06-15T06:17:36", "2024-06-15T06:43:12", "2024-06-15T07:08:48", "2024-06-15T07:34:24", "2024-06-15T08:00:00", "2024-06-15T08:25:36", "2024-06-15T08:51:12", "2024-06-15T09:16:48", "2024-06-15T09:42:24", "2024-06-15T10:08:00", "2024-06-15T10:33:36", "2024-06-15T10:59:12", "2024-06-15T11:24:48", "2024-06-15T11:50:24", "2024-06-15T12:16:00", "2024-06-15T12:41:36", "2024-06-15T13:07:12", "2024-06-15T13:32:48", "2024-06-15T13:58:24", "2024-06-15T14:24:00", "2024-06-15T14:49:36", "2024-06-15T15:15:12", "2024-06-15T15:40:48", "2024-06-15T16:06:24", "2024-06-15T16:32:00", "2024-06-15T16:57:36", "2024-06-15T17:23:12", "2024-06-15T17:48:48", "2024-06-15T18:14:24", "2024-06-15T18:40:00", "2024-06-15T19:05:36", "2024-06-15T19:31:12", "2024-06-15T19:56:48", "2024-06-15T20:22:24", "2024-06-15T20:48:00", "2024-06-15T21:13:36", "2024-06-15T21:39:12", "2024-06-15T22:04:48", "2024-06-15T22:30:24", "2024-06-15T22:56:00", "2024-06-15T23:21:36", "2024-06-15T23:47:12", "2024-06-16T00:12:48", "2024-06-16T00:38:24", "2024-06-16T01:04:00", "2024-06-16T01:29:36", "2024-06-16T01:55:12", "2024-06-16T02:20:48", "2024-06-16T02:46:24", "2024-06-16T03:12:00", "2024-06-16T03:37:36", "2024-06-16T04:03:12", "2024-06-16T04:28:48", "2024-06-16T04:54:24", "2024-06-16T05:20:00", "2024-06-16T05:45:36", "2024-06-16T06:11:12", "2024-06-16T06:36:48", "2024-06-16T07:02:24", "2024-06-16T07:28:00", "2024-06-16T07:53:36", "2024-06-16T08:19:12", "2024-06-16T08:44:48", "2024-06-16T09:10:24", "2024-06-16T09:36:00", "2024-06-16T10:01:36", "2024-06-16T10:27:12", "2024-06-16T10:52:48", "2024-06-16T11:18:24", "2024-06-16T11:44:00", "2024-06-16T12:09:36", "2024-06-16T12:35:12", "2024-06-16T13:00:48", "2024-06-16T13:26:24", "2024-06-16T13:52:00", "2024-06-16T14:17:36", "2024-06-16T14:43:12", "2024-06-16T15:08:48", "2024-06-16T15:34:24", "2024-06-16T16:00:00", "2024-06-16T16:25:36", "2024-06-16T16:51:12", "2024-06-16T17:16:48", "2024-06-16T17:42:24", "2024-06-16T18:08:00", "2024-06-16T18:33:36", "2024-06-16T18:59:12", "2024-06-16T19:24:48", "2024-06-16T19:50:24", "2024-06-16T20:16:00", "2024-06-16T20:41:36", "2024-06-16T21:07:12", "2024-06-16T21:32:48", "2024-06-16T21:58:24", "2024-06-16T22:24:00", "2024-06-16T22:49:36", "2024-06-16T23:15:12", "2024-06-16T23:40:48"], "xaxis": "x", "y": [2.3218959922790527, 2.3312800197601318, 2.2129147163657255, 2.163582698566707, 2.113913378377599, 2.195304699242115, 2.2397265546023846, 2.141464560050664, 2.13153126090765, 2.3158828113228083, 2.407156253233552, 2.2772283272480403, 2.1641889801175576, 2.2944461547411406, 2.258409460698526, 2.172294577886892, 2.2456796932965517, 2.246929693967104, 2.2293281331658363, 2.4183149656911533, 2.2337922958227305, 2.2802343741059303, 2.4050000007335957, 2.3258682202923207, 2.3194496040196384, 2.3413255824599157, 2.3819921761751175, 2.142609376460314, 2.2720076946111827, 2.210226560011506, 2.364968754351139, 2.2719062492251396, 2.2568593695759773, 2.3399375155568123, 2.3113412781367226, 2.1056015584617853, 2.27888976495097, 2.358661418824684, 2.3824297040700912, 2.136883719022884, 2.209124995395541, 2.3374615210753222, 2.2748515717685223, 2.2328425298525594, 2.193203119561076, 2.2001782868259636, 2.1906093806028366, 2.254798464072767, 2.179708659179567, 2.2527907153432682, 2.2131574867278574, 2.182539062574506, 2.238055105284443, 2.2725859377533197, 2.1403046902269125, 2.1290468648076057, 2.159823072873629, 2.5134206347995334, 2.1682812478393316, 2.1244843751192093, 2.211789069697261, 2.1320708736659975, 2.165890632197261, 2.218601556494832, 2.291409454946443, 2.1572790737299954, 2.1927999935150146, 2.101007857660609, 2.3569370179664433, 2.262844965439434, 2.2352187503129244, 2.0989055014032076, 2.135801574540517, 2.2344218492507935, 2.2556800041198732, 2.1369212612392396, 2.257877875830381, 2.2916153816076426, 2.293881893157959, 2.1787286784297737, 2.2223100773123807, 2.2671406269073486, 2.287007825449109, 2.163140630349517, 2.2871162743531457, 2.273867202922702, 2.2314409552596683, 2.231562504544854, 2.2734523784546625, 2.2467597011447875, 2.2423999841396625, 2.184511819223719, 2.188162783319636, 2.252335960045457, 2.183147289956263, 2.170921863988042, 2.114554664120078, 2.1190387315528336, 2.194148428738117, 2.3339457401009494, 2.2246171943843365, 2.1415547020733356, 2.181968514374861, 2.266820279881358, 2.223039072006941, 2.3223359966278077, 2.2296062942564956, 2.16088976634769, 2.1915511747044842, 2.183015752026415, 2.298359375447035, 2.3426230925780076, 2.2092063483737765, 2.1584453098475933, 2.2480157660687063, 2.096449613571167, 2.2344062495976686, 2.2501406371593475, 2.1975118231585644, 2.1788449527681335, 2.1882755906563105, 2.21124408939692, 2.1671406254172325, 2.2697578240185976, 2.3204921931028366, 2.299687499180436, 2.291999994322311, 2.241999997867374, 2.219414059072733, 2.2414843812584877, 2.278364334919656, 2.242534886958987, 2.1956640649586916, 2.2084961974893815, 2.3241171799600124, 2.270046880468726, 2.2242558057918105, 2.275132816284895, 2.2340787320625126, 2.1789140589535236, 2.1303359493613243, 2.1715625133365393, 2.197484366595745, 2.274789063259959, 2.232362212158564, 2.307601572945714, 2.21406300612322, 2.193703133612871, 2.2250866176575186, 2.1807165352378304, 2.1098527021186295, 2.1490000132471323, 2.215806455381455, 2.3345891579177027, 2.335612404254056, 2.1643750183284283, 2.1499140467494726, 2.0842077090190005, 2.232429686933756, 2.277401576830646, 2.2026744147603825, 2.1334687527269125, 2.285480624945589, 2.193307089993334, 2.224141740423488, 2.2731627771096634, 2.179565886194392, 2.0006328001618385, 2.1440515370713067], "yaxis": "y", "type": "scatter"}], "layout": {"template": {"data": {"scatter": [{"type": "scatter"}]}}, "xaxis": {"anchor": "y", "domain": [0.0, 1.0], "title": {"text": "Date"}}, "yaxis": {"anchor": "x", "domain": [0.0, 1.0], "title": {"text": "Arrival time within slot"}}, "legend": {"title": {"text": "topic"}, "tracegroupgap": 0}, "margin": {"t": 60}, "title": {}}}