AutoNATv2 implementation for rust-libp2p

[umgefahren]

Open Grant Proposal: AutoNATv2 implementation for rust-libp2p

Proposer: @umgefahren - Hannes Furmans

Proposal Category: core-dev

Technical Sponsor: @mxinden - Max Inden

Do you agree to open
source all work you do on behalf of this RFP and dual-license under MIT and APACHE2 licenses?: Yes

Project description

Since libp2p is adopting the second generation of AutoNAT, namely AutoNATv2, and there is already a go-libp2p implementation under way (libp2p/go-libp2p#2469), it's time to bring those changes to rust-libp2p.

While addressing these new challenges it might be a good time to fix some low-hanging fruits and legacy code on the way, if there is time.

Project value

Implementing these changes is important to keep rust-libp2p up-to-date with the current libp2p spec. The new AutoNAT solves problems of the old one, namely being able to test reachability on one address, bringing value to the libraries users, for instance Iroh. It's also worth pointing out that the the protocol has decreased complexity, which will make maintenance easier.

The implementation will require some changes to core traits (in simple terms Rust interfaces), which means design, migration and coordination time is necessary.

Deliverables

The deliverables are many pull requests to libp2p/rust-libp2p, introducing and merging the promised new features.

If requested there will also be a report at the end reviewing difficulties and lessons learned on the road, to make it easier for the other libp2p implementations to adopt the changes.

Development Roadmap

All of the implementation work is done by Hannes Furmans with technical and project guidance and advice from Max Inden.

Since it's quite hard to estimate how difficult implementing these changes turns out, the milestones are very likely be subject to change.

25 Sep 2023: Proposed date to start

Number	Milestone	Hours	Cost
0	Step 1 of PoC: Adjust Transport::dial	57,33	5.733
1	Step 2 of PoC: Implement AutoNATv2	57,33	5.733
2	Adjust all Transport and NetworkBehaviour implementations	57,33	5.733
	Total	172	17.200

Hannes will work on a 10-hour-per-week basis, the project will therefore take about 17 weeks in total.

Hannes will do time tracking and state the time it took to complete a milestone. If there is time left, Hannes will work on different issues around the project, filling the remaining time of the grant.

Total Budget Requested

17200 US-$

No VAT charged.

The grant proposer would suggest a pay-by-milestone deal, since the already described issues in estimation. However, a change in cost won't affect an ongoing milestone. The budget stays predictable.

Maintenance and Upgrade Plans

Since the project is about software there will be maintenance costs. The grant proposer will take part in maintenance efforts, balancing current and future obligations. The grant proposer is willing to commit to maintenance efforts, given apporpriate compensation and bandwith.

Team

Team Members

• Hannes Furmans (Software Developer) -
  https://github.com/umgefahren
• Max Inden (Protocol Labs advisor) - https://max-inden.de

Team Website

None

Relevant experience

Hannes is a frequent contributor to rust-libp2p. He wrote the Tor transport for rust-libp2p and also build software using rust-libp2p.

[umgefahren]

What is the status of my grant?

[mxinden]

@eshon can you provide an update here or point @umgefahren to a place or person to reach out to?

[umgefahren]

@eshon can you provide an update here or point @umgefahren to a place or person to reach out to?

No need. I already reached out to her yesterday and got the latest information. I just posted that comment, because in the README it states you can check in after 30 days.

[ErinOCon]

Hi @umgefahren, thank you for your proposal and for your patience with our review! We would like to move your proposal forward to the next steps in our process. We will send an email with further details.

[umgefahren]

Hi @ErinOCon
I'm really looking forward to that email 🚀 😄

[ErinOCon]

HI @umgefahren, can you confirm the preferred email address for this communication? Many thanks!

[umgefahren]

Sure: [email protected]

[mxinden]

@umgefahren @ErinOCon what is the current status on the grant?

[mxinden]

@umgefahren @ErinOCon @eshon friendly ping. What is the status on the grant? Do I understand correctly that this is not going to happen?

[umgefahren]

Hi @mxinden,
I honestly don't know. I got a confirmation and the notice that there will be an Open Source Agreement for me to sign soon. It has been around two months since and I haven't heard anything. Exam season is starting soon, so my focus shifted away for the moment anyway, although this is really disappointing.

[ErinOCon]

HI @umgefahren, I am so sorry for the delay! I have just sent you an email. Let me know if you have any questions.

[mxinden]

Administrative

What is the current status of this grant from an administrative side @umgefahren and @ErinOCon? What is the current planned timeline?

Technical

Since this grant was written a couple of things changed.

• ProbeLab started a new effort, namely to rewrite https://github.com/libp2p/go-libp2p-kad-dht/. You can find the in-progress work on https://github.com/plprobelab/go-kademlia. If I understand correctly the major objective is to write a generic Kademlia implementation, useful beyond IPFS, eventually used by IPFS among other projects.
• I am not aware of anyone directly working on making the IPFS DHT more privacy preserving right now. As far as I am aware the rewrite (above) is seen as an intermediate goal to be achieved in order to enable the privacy preserving work in the future.

//CC @guillaumemichel @dennis-tra and @yiannisbot from the ProbeLab team

With the above, I am hesitant to have rust-libp2p be the pioneer on the privacy preserving DHT front. Not that I have doubt in you @umgefahren tackling this, but that I don't have the capacity to mentor you without the support of some other team (e.g. ProbeLab) providing a reference implementation and deployment.

I don't think this should invalidate the grant, nor do I think we need to restart the grant from scratch. Instead I am proposing the following:

• Continue as planned with the above described "Value of fixing low-hanging fruits".
• Choose an additional project on rust-libp2p. Suggestions from my end:
  • With rust-libp2p's move to upstream quinn () we can now integrate hyperiums WebTransport implementation (https://github.com/hyperium/h3) to add libp2p WebTranport support to rust-libp2p. We already have a WASM WebTransport implementation.
  • Implement the upcoming AutoNATv2 protocol in rust-libp2p.
  • Alternative suggestions welcome @umgefahren.

@umgefahren let me know what you think.

@ErinOCon let me know if this change in technical direction poses any issues.

[umgefahren]

Administrative

Ironically I received the email to e-sign the contract 2 hours ago (at the time of writing). The contract also stated that work would start on the 28th of August. That was agreed upon since I'm currently in the middle of exam season and work should start after that.

Technical

• I wasn't aware of the ProbeLab effort. Truth be told; I was working on other projects since this process started, nevertheless I would still like to work on rust-libp2p.
• If I recall correctly the initial motivation for this project was the ongoing implementation of Double Hashing in the Go implementation by ChainSafe and my understanding was that all libp2p projects were to introduce Double Hashing in the near term future. Including Double Hashing into the IPFS Spec has also stalled since March (IPIP-373: Double Hash DHT Spec ipfs/specs#373). The corresponding PRs, although inactive since January, were:
  • prefix lookup implementation ChainSafe/go-libp2p-kad-dht#2
  • provider record encryption ChainSafe/go-libp2p-kad-dht#3
    I don't know what happened to the project, maybe @mxinden knows more.

With the above, I am hesitant to have rust-libp2p be the pioneer on the privacy preserving DHT front. Not that I have doubt in you @umgefahren tackling this, but that I don't have the capacity to mentor you without the support of some other team (e.g. ProbeLab) providing a reference implementation and deployment.

Since priorities have shifted away from the greater project, I understand that this grant, or at least a major part has outlived its roots. Considering how long it took to arrive at a point where I could actually sign the contract, this is somewhat frustrating.

I would still like to work on rust-libp2p. So I'm clearly in favor of choosing a different project in rust-libp2p.

From the brief look I took, I would prefer implementing the AutoNATv2 protocol, but WebTransport is also fine. Another possible project could be general improvements to our DHT implementations, I'm thinking of all the performance optimizations that are already implemented in go-libp2p-kad-dht (or the new replacement). On either project, I would have to talk to @mxinden, to figure out how the milestones need to be modified.

If we shift the focus, I would be very happy if we could still make the targeted start of work day (28th of August), since I've already included the grant in my plans for the next semester. If that's not possible, we will find a different solution.

Considering how necessary the shift is, I'm also very glad that @mxinden made me/us aware of the changes happening in the project. It came just in time.

@ErinOCon please let me know what I can do to still make this grant possible and shift directions.

[ErinOCon]

Thank you for sending notice of the updates, @mxinden! @umgefahren, we would be very happy to review an updated proposal for your work on rust-libp2p. Please take the time you need to review further.

[umgefahren]

I will evaluate with @mxinden and will submit an updated proposal as soon as possible.

[mxinden]

• provider record encryption ChainSafe/go-libp2p-kad-dht#3
  I don't know what happened to the project, maybe @mxinden knows more.

I am not aware of any progress on it. If I understand correctly (//CC @dennis-trautwein) the re-write in https://github.com/plprobelab/go-kademlia is prioritized.

I would prefer implementing the AutoNATv2 protocol

Fine by me. Though needs some refactorings within libp2p-core see libp2p/rust-libp2p#3900 (comment) and libp2p/rust-libp2p#4226.

Another possible project could be general improvements to our DHT implementations, I'm thinking of all the performance optimizations that are already implemented in go-libp2p-kad-dht (or the new replacement).

Fine by me as well. Though would need to focus on specific optimizations. I would prefer AutoNATv2 or WebTransport due to higher impact. That said, the initially proposed clean-ups are always appreciated.

@umgefahren maybe you want to join the open maintainers call tomorrow to discuss this in more depth? libp2p/rust-libp2p#4276

[guillaumemichel]

Quick update concerning the Go Kademlia Double Hash implementation:

• The work done by ChainSafe is unfortunately not usable
• ProbeLab has taken up the implementation after deciding to stop working with ChainSafe
• Because of the state of go-libp2p-kad-dht, the implementation has shown itself trickier than expected.
• We are currently working to refactor go-libp2p-kad-dht using go-kademlia.

Our roadmap is currently as follow (it is subject to change)

• Ship the refactored Go Kademlia implementation
• Write Specs and implement Composable Kademlia enabling easier protocol upgrades
• Implement the Double Hash Kademlia in Go, and merge it using Composable Kademlia (unlikely to happen before 2024Q1)

The Double Hashing DHT spec is still expected to evolve (even though it isn't currently being working on) and will certainly depend on the (still non-existant) Composable Kademlia Spec. I think it makes sense to delay the rust Double Hash DHT implementation after we have locked both the Composable Kademlia and Double Hash DHT specs, and ideally after the reference implementation in Go is ready.

@umgefahren I am sorry for the change of plans. IMO it would make sense, if possible, to change the scope of this project.

[mxinden]

Thank you @guillaumemichel for the detailed update here. Much appreciated!

---

• Implement the upcoming AutoNATv2 protocol in rust-libp2p.

For what it is worth @sukunrt started an implementation of the AutoNAT v2 specification for go-libp2p here libp2p/go-libp2p#2469.

[umgefahren]

Hi everyone and thanks for the patience. The last three exams were difficult and took all of my time.

I spoke with @mxinden and @thomaseizinger in the rust-libp2p open maintainers call. The specific notes can be found here: rust-libp2p open maintainers call notes

We agreed on rescoping the grant, with the same budgeted time, but different work. Max and Thomas were positive that the new scope will fill the grant. If not, I will do time tracking and fill the rest of the grant with work. There is no shortage of that in rust-libp2p.

Here is what we've agreed on regarding the new scope:
Instead of implementing Kademlia Double Hashing, I will implement the upcoming AutoNATv2.

For what it is worth @sukunrt started an implementation of the AutoNAT v2 specification for go-libp2p here libp2p/go-libp2p#2469.

Very basically, this is important, because rust-libp2p should keep up with go-libp2p and implement the new protocol as well. This is the same reason why implementing Kadmelia Double Hashing (was) valuable.

The deliverables are once again a series of Pull Requests to rust-libp2p, a milestone is considered complete when the PR is approved by @mxinden.

Development Roadmap

Number	Milestone	Hours	Cost
0	Step 1 of PoC: Adjust Transport::dial	57,33	5.733
1	Step 2 of PoC: Implement AutoNATv2	57,33	5.733
2	Adjust all Transport and NetworkBehaviour implementations	57,33	5.733
	Total	172	17.200

Hannes will work on a 10-hour-per-week basis, the project will therefore take about 17 weeks in total.

Everything else stays the same. If requested, I can subdivide these milestones.

The hole project includes a lot of design work and with that a lot of coordination with the rest of the team.

I hope this updated proposal suffices to update the OSA and start work soon. The new milestones are now roughly 6 weeks apart, similar to the OSA.

It's probably not possible to start work in six days. I apologize for that. I hope we can start soon anyway.

Have a great weekend everyone :)

[ErinOCon]

Hi @umgefahren, thank you for the provided update! Our team is currently reviewing and will be in touch.

[ErinOCon]

Hi @umgefahren, these changes are in good order to proceed with our granting process. For next steps, can you update the original application? The updated project title, value, description, and milestones will support our contract review.

[umgefahren]

I'm sorry I have missed this commend. I have updated the project title, value, description and milestones. If requested I can make the milestones smaller, but these were in accordance with Max.

[ErinOCon]

Thank you, @umgefahren!