From 9e323d4f17df6d558836a61b4395cb05aa2691a5 Mon Sep 17 00:00:00 2001 From: hangleang Date: Thu, 4 Jul 2024 12:16:20 +0700 Subject: [PATCH 1/4] propose a project on PeerDAS in Grandine --- projects/peerdas-grandine.md | 64 ++++++++++++++++++++++++++++++++++++ 1 file changed, 64 insertions(+) create mode 100644 projects/peerdas-grandine.md diff --git a/projects/peerdas-grandine.md b/projects/peerdas-grandine.md new file mode 100644 index 00000000..d9253c8d --- /dev/null +++ b/projects/peerdas-grandine.md @@ -0,0 +1,64 @@ +# PeerDAS in Grandine + +PeerDAS specs is still maturing, so there are a lot of changes and improvements that need to be implemented in Grandine. + +*Written by Hangleang ([@HangleangS](https://twitter.com/HangleangS))* + +## Motivation + +The [recent blob fee market issues](https://x.com/mcutler/status/1803876002734735381) went crazy when too many type-3 transactions *(blob-carrying-transaction)* were competing to submit onchain, which is a clear evidence that the need for more scalability is essential for handling that much demands from L2-rollups. + +*Porting from motivation in [EIP-7594](https://eips.ethereum.org/EIPS/eip-7594)* + +> DAS is a method of scaling data availability beyond the levels of EIP-4844 by not requiring all nodes to download all data while still ensuring that all of the data has been made available. +> +> Providing additional data availability helps bring scale to Ethereum users in the context of layer 2 systems called “roll-ups” whose dominant bottleneck is layer 1 data availability. + +## Project description + +*Abstraction from [EIP-7594](https://eips.ethereum.org/EIPS/eip-7594)* +> PeerDAS (Peer Data Availability Sampling) is a networking protocol that allows beacon nodes to perform data availability sampling (DAS) to ensure that blob data has been made available while downloading only a subset of the data. PeerDAS utilizes gossip for distribution, discovery for finding peers of particular data custody, and peer requests for sampling. + +The proposed solution is to continually improve existing PeerDAS implementation, which is done on major functionality by Grandine team. Since the specs is involving on different layer stack, it's better to divide tasks among fellows, we will have a person or two to porting on `libp2p` gossipsub and peer sampling *(networking)*, another person to handle updating core protocol to support lastest specs and ensure interop with other clients implementation *(core)*, while getting feedbacks and mentoring from Grandine team when we got stuck on the road. + +## Specification + +- For core protocol, we would be using [`das-core`](https://github.com/ethereum/consensus-specs/blob/dev/specs/_features/eip7594/das-core.md) specs as our compass, and sync with existing implementation. +- For p2p networking, ... +- For kzg commitment cryptography, ... + +## Roadmap + +Since a few fellows are working on the same project, we will be dividing the work among ourselves. We got a draft plan from [mentors](#Mentors): +- Update the `das` branch to support latest `PeerDAS` spec +- Update `eth2_libp2p` library by backporting changes from [lighthouse](https://github.com/sigp/lighthouse/tree/das-devnet-1/beacon_node/lighthouse_network) to [eth2_libp2p](https://github.com/grandinetech/eth2_libp2p/tree/eip_7594) +- Run `Kurtosis` interop with other three clients and identify/fix incompatibility issues + +## Possible challenges + +- **Limited Resources and Docs:** Grandine is a new implementation of consensus client with a few contributors/developers, but a handful of packages inside a single monorepo. By that, developers has no time to write a proper developer documentation. So for external contributors like us, we have to spend some times to get familiar with which package does what. + +## Goal of the project + +The goal of the project is to implement PeerDAS in Grandine, improve the existing implementation, optimize for better performance and security of the client, and ensure reliability of the network and interoperability between others client implementations. + +Beside implementation, battle tests need to be addressed against unexpected scenarios. So, it is essential to implement tests case and benchmarking those metrics as well. + +## Collaborators + +### Fellows + +- [Hangleang SUN](https://github.com/hangleang) +- ... + +### Mentors + +- [Saulius Grigaitis](https://github.com/sauliusgrigaitis) +- ... + +## Resources + +- [Grandine Repo](https://github.com/grandinetech/grandine/tree/das) +- [PeerDAS Specs](https://github.com/ethereum/consensus-specs/tree/dev/specs/_features/eip7594) +- [eth2_libp2p Repo](https://github.com/grandinetech/eth2_libp2p/tree/eip_7594) +- [C-binding KZG Library](https://github.com/ethereum/c-kzg-4844/tree/das) From 46b0adc886f39d2989b014b0b931be870a168837 Mon Sep 17 00:00:00 2001 From: hangleang Date: Sat, 6 Jul 2024 09:03:58 +0700 Subject: [PATCH 2/4] update proposal --- projects/peerdas-grandine.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/projects/peerdas-grandine.md b/projects/peerdas-grandine.md index d9253c8d..ef22ac33 100644 --- a/projects/peerdas-grandine.md +++ b/projects/peerdas-grandine.md @@ -1,9 +1,11 @@ -# PeerDAS in Grandine +# PeerDAS improvements in Grandine PeerDAS specs is still maturing, so there are a lot of changes and improvements that need to be implemented in Grandine. *Written by Hangleang ([@HangleangS](https://twitter.com/HangleangS))* +*Thanks to [Saulius Grigaitis](https://github.com/sauliusgrigaitis) for feedback and review* + ## Motivation The [recent blob fee market issues](https://x.com/mcutler/status/1803876002734735381) went crazy when too many type-3 transactions *(blob-carrying-transaction)* were competing to submit onchain, which is a clear evidence that the need for more scalability is essential for handling that much demands from L2-rollups. @@ -24,8 +26,8 @@ The proposed solution is to continually improve existing PeerDAS implementation, ## Specification - For core protocol, we would be using [`das-core`](https://github.com/ethereum/consensus-specs/blob/dev/specs/_features/eip7594/das-core.md) specs as our compass, and sync with existing implementation. -- For p2p networking, ... -- For kzg commitment cryptography, ... +- For p2p networking, a fellow is working on backporting latest commits into [eth2_libp2p](https://github.com/grandinetech/eth2_libp2p/tree/eip_7594) compatible types in [grandine](https://github.com/grandinetech/grandine/tree/das), which maintained by Grandine team. +- For kzg commitment cryptography, we would be using [c-binding KZG lib](https://github.com/ethereum/c-kzg-4844/tree/das) maintained by EF team, which is ready to use in the mean time, then switch to [rust-kzg](https://github.com/grandinetech/rust-kzg) once ready, maintained by Grandine team. a few fellow are also working on porting functionalities to the lib. ## Roadmap @@ -49,16 +51,14 @@ Beside implementation, battle tests need to be addressed against unexpected scen ### Fellows - [Hangleang SUN](https://github.com/hangleang) -- ... ### Mentors - [Saulius Grigaitis](https://github.com/sauliusgrigaitis) -- ... ## Resources - [Grandine Repo](https://github.com/grandinetech/grandine/tree/das) - [PeerDAS Specs](https://github.com/ethereum/consensus-specs/tree/dev/specs/_features/eip7594) - [eth2_libp2p Repo](https://github.com/grandinetech/eth2_libp2p/tree/eip_7594) -- [C-binding KZG Library](https://github.com/ethereum/c-kzg-4844/tree/das) +- [c-binding KZG Library](https://github.com/ethereum/c-kzg-4844/tree/das) \ No newline at end of file From 5a288ced9c614797a62895e339e3577e7d95379d Mon Sep 17 00:00:00 2001 From: hangleang Date: Mon, 15 Jul 2024 10:31:45 +0700 Subject: [PATCH 3/4] add rahul as collaborator --- projects/peerdas-grandine.md | 1 + 1 file changed, 1 insertion(+) diff --git a/projects/peerdas-grandine.md b/projects/peerdas-grandine.md index ef22ac33..00a1fa63 100644 --- a/projects/peerdas-grandine.md +++ b/projects/peerdas-grandine.md @@ -51,6 +51,7 @@ Beside implementation, battle tests need to be addressed against unexpected scen ### Fellows - [Hangleang SUN](https://github.com/hangleang) +- [Guha Rahul](https://github.com/guha-rahul) ### Mentors From 0c9923ddd283d28f060d33ac81bb6a02a6f7f825 Mon Sep 17 00:00:00 2001 From: hangleang Date: Thu, 18 Jul 2024 10:37:14 +0700 Subject: [PATCH 4/4] update proposal, add challenges --- projects/peerdas-grandine.md | 29 +++++++++++++++++------------ 1 file changed, 17 insertions(+), 12 deletions(-) diff --git a/projects/peerdas-grandine.md b/projects/peerdas-grandine.md index 00a1fa63..4a1d011e 100644 --- a/projects/peerdas-grandine.md +++ b/projects/peerdas-grandine.md @@ -1,8 +1,8 @@ # PeerDAS improvements in Grandine -PeerDAS specs is still maturing, so there are a lot of changes and improvements that need to be implemented in Grandine. +**TLDR**: link to our [slides proposal](https://docs.google.com/presentation/d/1_mKAAYlF0kKhDwaJ67o5_ki4_aSYg4SSGwiMuZD9sgg/edit?usp=sharing) for our motivation on the project. -*Written by Hangleang ([@HangleangS](https://twitter.com/HangleangS))* +PeerDAS specs is still maturing, so there are a lot of changes and improvements that need to be implemented in Grandine. *Thanks to [Saulius Grigaitis](https://github.com/sauliusgrigaitis) for feedback and review* @@ -21,30 +21,34 @@ The [recent blob fee market issues](https://x.com/mcutler/status/180387600273473 *Abstraction from [EIP-7594](https://eips.ethereum.org/EIPS/eip-7594)* > PeerDAS (Peer Data Availability Sampling) is a networking protocol that allows beacon nodes to perform data availability sampling (DAS) to ensure that blob data has been made available while downloading only a subset of the data. PeerDAS utilizes gossip for distribution, discovery for finding peers of particular data custody, and peer requests for sampling. -The proposed solution is to continually improve existing PeerDAS implementation, which is done on major functionality by Grandine team. Since the specs is involving on different layer stack, it's better to divide tasks among fellows, we will have a person or two to porting on `libp2p` gossipsub and peer sampling *(networking)*, another person to handle updating core protocol to support lastest specs and ensure interop with other clients implementation *(core)*, while getting feedbacks and mentoring from Grandine team when we got stuck on the road. +The proposed solution is to continually improve existing PeerDAS implementation, which is done on major functionality by Grandine team. Since the specs is involving on different layer stack, it's better to divide tasks among fellows, we will have a person to porting on `libp2p` gossipsub and peer sampling *(networking)*, another person to handle updating core protocol to support lastest specs and ensure interop with other clients implementation *(core)*, while getting feedbacks and mentoring from Grandine team when we got stuck on the road. ## Specification -- For core protocol, we would be using [`das-core`](https://github.com/ethereum/consensus-specs/blob/dev/specs/_features/eip7594/das-core.md) specs as our compass, and sync with existing implementation. -- For p2p networking, a fellow is working on backporting latest commits into [eth2_libp2p](https://github.com/grandinetech/eth2_libp2p/tree/eip_7594) compatible types in [grandine](https://github.com/grandinetech/grandine/tree/das), which maintained by Grandine team. -- For kzg commitment cryptography, we would be using [c-binding KZG lib](https://github.com/ethereum/c-kzg-4844/tree/das) maintained by EF team, which is ready to use in the mean time, then switch to [rust-kzg](https://github.com/grandinetech/rust-kzg) once ready, maintained by Grandine team. a few fellow are also working on porting functionalities to the lib. +- For core protocol, we would be using [`das-core`](https://github.com/ethereum/consensus-specs/blob/dev/specs/_features/eip7594/das-core.md) specs as our compass, update existing implementation, and sync with other clients implementation. + +- For p2p networking, we are backporting latest commits into [eth2_libp2p](https://github.com/grandinetech/eth2_libp2p/tree/eip_7594) compatible types in [grandine](https://github.com/grandinetech/grandine/tree/das). + +- For kzg commitment cryptography, we would be using [c-kzg lib](https://github.com/ethereum/c-kzg-4844/tree/das) maintained by EF team, which is ready to use in the meantime, then switch to [rust-kzg](https://github.com/grandinetech/rust-kzg) once ready, maintained by Grandine team. Other folks are working on porting functionalities to the library. ## Roadmap Since a few fellows are working on the same project, we will be dividing the work among ourselves. We got a draft plan from [mentors](#Mentors): -- Update the `das` branch to support latest `PeerDAS` spec -- Update `eth2_libp2p` library by backporting changes from [lighthouse](https://github.com/sigp/lighthouse/tree/das-devnet-1/beacon_node/lighthouse_network) to [eth2_libp2p](https://github.com/grandinetech/eth2_libp2p/tree/eip_7594) -- Run `Kurtosis` interop with other three clients and identify/fix incompatibility issues +- Updating `PeerDAS` implementation to the latest specs +- Backporting `eth2_libp2p` library from [lighthouse](https://github.com/sigp/lighthouse/tree/das-devnet-1/beacon_node/lighthouse_network) +- Testing, Debugging `kurtosis` local devnets +- Ensure interop with other clients implementation ## Possible challenges -- **Limited Resources and Docs:** Grandine is a new implementation of consensus client with a few contributors/developers, but a handful of packages inside a single monorepo. By that, developers has no time to write a proper developer documentation. So for external contributors like us, we have to spend some times to get familiar with which package does what. +- **Limited Resources and Docs:** Grandine is a new consensus client implementation with a few contributors/core developers handle a handful of packages inside a single monorepo. They just open their source code in early 2024, so developers has no time to write a proper developer documentation. Their focus is on internal development, so for external contributors like us, we have to spend some times to get familiar with which package does what. A fellow is working on documentation and testing, we will collab with him on this side as well. +- **Devnet Monitoring and Debugging:** We have no experience at all on these aspects, it might be abit hard for us to understand the logs. Even if we see some errors/warning, we might don't know where it came from, and what its cause, and we are not sure if it's working or not even thought there is no errors. We might take some time to understand the logs, get used to it. ## Goal of the project The goal of the project is to implement PeerDAS in Grandine, improve the existing implementation, optimize for better performance and security of the client, and ensure reliability of the network and interoperability between others client implementations. -Beside implementation, battle tests need to be addressed against unexpected scenarios. So, it is essential to implement tests case and benchmarking those metrics as well. +Beside implementation, battle tests need to be addressed against unexpected scenarios. So, it is essential to implement tests case and benchmarking those metrics as well. A fellow is working on metrics and benchmarking, we will collab with her on these aspects. ## Collaborators @@ -62,4 +66,5 @@ Beside implementation, battle tests need to be addressed against unexpected scen - [Grandine Repo](https://github.com/grandinetech/grandine/tree/das) - [PeerDAS Specs](https://github.com/ethereum/consensus-specs/tree/dev/specs/_features/eip7594) - [eth2_libp2p Repo](https://github.com/grandinetech/eth2_libp2p/tree/eip_7594) -- [c-binding KZG Library](https://github.com/ethereum/c-kzg-4844/tree/das) \ No newline at end of file +- [c-binding KZG Library](https://github.com/ethereum/c-kzg-4844/tree/das) +- [kurtosis local devnets](https://ethpandaops.io/posts/kurtosis-deep-dive/) \ No newline at end of file