Date: 2022-10-20
Status: Accepted
Notes: Internal test-plans stakeholders have aligned on this roadmap. Please add any feedback or questions in:
https://github.com/libp2p/test-plans/issues/58
- About the Roadmap
- 🛣️ Milestones
- 📖 Appendix
We, the maintainers, are committed to upholding libp2p's shared core tenets and ensuring libp2p implementations are: Secure, Stable, Specified, and Performant.
This roadmap is complementary to those of go-libp2p, rust-libp2p, and js-libp2p.
It aims to encompass the stability and performance tenets of the libp2p team. Projects outlined here are shared priorities of the different implementations.
This document consists of two sections: Milestones and the Appendix
Milestones is our best educated guess (not a hard commitment) around when we plan to ship the key features. Where possible projects are broken down into discrete sub-projects e.g. project "A" may contain two sub-projects: A.1 and A.2
A project is signified as "complete" once all of it's sub-projects are shipped.
The Appendix section describes a project's high-level motivation, goals, and lists sub-projects.
Each Appendix header is linked to a GitHub Epic. Latest information on progress can be found in the Epics and child issues.
Projects are listed in descending priority.
Why: libp2p supports a variety of transport protocols, muxers, & security protocols across implementations in different languages. Until we actually test them together, we can't guarantee implementation interoperability. We need to ensure and demonstrate that: interoperable features work with each other as expected and we don't introduce degradations that break interoperability in new releases.
Goal: libp2p implementers run tests across permutations of libp2p implementations, versions, and supported transports, muxers, and security protocols. Implementers and users can reference a single website with a dashboard to view the Pass/Fail/Implemented/Not Implemented results of multi-dimensional tests.
This effort depends on Testground Milestone 1
How:
Enable test-plans maintainers to define a configuration (of libp2p impls, versions, transports, expected RTT), run Testground tests per configuration, and retrieve resulting data in a standard format. The test case results can be formatted as a "dashboard" (simple Markdown table of Pass/Fail results.)
Using tooling from A.1, all features of go-libp2p, rust-libp2p, and js-libp2p that should be interoperable are tested (i.e. transports (TCP, QUIC, WebRTC, WebTransport), multiplexers (mplex, yamux), secure channels (TLS, Noise), etc.) across versions.
Features currently in development across implementations (like WebRTC in go-libp2p and rust-libp2p, or QUIC & TLS in rust-libp2p) are not merged without at least manually running these test suites.
Test suites are run in libp2p/test-plans CI and before releasing a version of go-libp2p, rust-libp2p, and js-libp2p (GitHub workflow added so that these suites run against the master branch on a nightly basis (updating the status check.))
Note:
- Dependency on C.1 to run node.js-libp2p in Testground.
- Dependency on testground/Investigate browser test support to run interoperability test for js-libp2p WebRTC against Go and Rust.
A comprehensive and canonical dashboard is generated and hosted in a publicly discoverable site that displays latest test suite results (Pass/Fail/Implemented/Not Implemented/Unimplementable) from a nightly CI run. All permutations of libp2p implementations, versions, and supported transports, muxers, & security protocols should be visible.
An enhancement of A.1 to make it easier for users and implementers to see the full state of libp2p interoperability.
libp2p/test-plans maintainers have a straightforward way to track the test suite stability and performance.
- We can track the status of every test combination stability from the interop project itself
- We can easily measure the consequence (improvements) of a pull request to the libp2p/interop repository
- We are alerted when an interop test starts failing on one of our client repositories, and we can dispatch the alert to repo maintainers.
We have an explicit, working, Design Process for adding new tests
- The design is documented in
./DESIGN.md. - The design is followed by the team when we add new features.
- There is a clear path when it comes to testing new features. This might mean testing multiple
mastersagainst each other.
We have ported the tests from libp2p/interop
- This repository implement tests
connect,dht,pubsub(ref) - At of writing (2022-09-27), it is disabled in
go-libp2p(ref), and it is used injs-libp2p(ref).
Why: For libp2p to be competitive, it needs to delivers comparable performance to widely used protocols on the internet, namely HTTP/2 and HTTP/3.
Goal: We have a test suite that runs libp2p transfers between nodes located at different locations all over the world, proving that libp2p is able to achieve performance on par with HTTP. The test suite is run on a continuous basis and results are published to a public performance dashboard.
- Benchmark go-libp2p, rust-libp2p, and go-libp2p
- Specifically add js-libp2p-transfer-performance as a test-plan and CI job to benchmark transfer times across releases to catch issues like #1342
- (Dependency: remote machines need Nix installed)
Benchmarking using first class support for remote runners (using remote:exec) in Testground
Why: Having interoperability tests with lots of transports, encryption mechanisms, and stream muxers is great. However, we need to stay backwards-compatible with legacy libp2p releases, with other libp2p implementations, and less advanced libp2p stacks.
Goal: Expand beyond unit tests and have expansive test-plan coverage that covers all protocols.
This effort depends on Testground Milestone 6
Test js-libp2p DHT Server Mode at scale (testbed of at least >20 nodes; ideally 100/1000+) in Testground Depends on C.1 Relates to Testground Milestone 4 (for large scale tests.)
Depends on testground/NAT and/or firewall support
Depends on testground/NAT and/or firewall support
Depends on testground/Network Simulation Fixes