Rendezvous: self registration?

[odesenfans]

Hello,

I'm improving a network of P2P nodes by moving the P2P stack to rust-libp2p. We have a relatively small number of nodes (~80). These nodes can be considered as always up (99%+). They currently discover each other by using a few bootstrap nodes and then advertising their existence through gossipsub. I'm thinking of moving that logic to rendezvous. Here is my design so far:

• I want applications to be able to use each node as a bootstrap node. All nodes should be considered equal for most purposes.
• I want each node to have a full view of the network. At this size, it seems reasonable.
• I don't want to create separate bootstrap nodes. The rendezvous server should be an integral part of each node ideally.

I've been struggling with the last point, as it requires the node to, well, self-register (otherwise it won't appear during discovery, despite being a perfectly valid node). I've successfully implemented a network behaviour which composes rendezvous::client::Behaviour, rendezvous::server::Behaviour and Identify. My initial strategy, inspired by the register_identify example, was to call rendezvous::client::Behaviour::register once the identify protocol got me an observed address for the local node. However, nothing happens when I do this.

1. What am I doing wrong?
2. Is this strategy even viable altogether?
3. If any, is there a peer discovery method that would suit this kind of use case better?

[thomaseizinger]

I think what is happening here is that you effectively try to dial yourself which doesn't work. In principle though, there is nothing wrong with the idea but the current implementation doesn't support it :)

What we'd have to do is expose a function on rendezvous::Behaviour::server to manually add entries to the underlying Registrations struct. Probably all that is necessary is that we add:

Index: protocols/rendezvous/src/server.rs
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/protocols/rendezvous/src/server.rs b/protocols/rendezvous/src/server.rs
--- a/protocols/rendezvous/src/server.rs	(revision 864237fa1855eb48859f9b9d7af1c68f660308a9)
+++ b/protocols/rendezvous/src/server.rs	(date 1662626496907)
@@ -79,6 +79,10 @@
             registrations: Registrations::with_config(config),
         }
     }
+
+    pub fn registrations_mut(&mut self) -> &mut Registrations {
+        &mut self.registrations
+    }
 }
 
 #[derive(Debug)]

This would allow you to call Registrations::add.

Allow me to add a more general note regarding discovery:

I don't know what your threat-model is but a common one in discovery protocols is censorship-resistence. It is not bullet-proof but the current design of the rendezvous protocol allows you to run a standalone rendezvous node that works across domains, i.e. it is not just useful for your network but in principle for any network that runs on top of libp2p. As such, an entity completely unrelated to you could operate this rendezvous node and offer it is a public service. This makes it difficult to censor / take down your network because the attacker first needs to take down the completely unrelated rendezvous node. If your application has support for multiple rendezvous nodes, this gets even more difficult because they have to repeat the process N times.

Having every node be a rendezvous server doesn't make a whole lot of sense in my opinion because it doesn't solve the problem of "how do I make the first connection?". Additionally, if the discovery protocol should be an integral part of your and only your nodes, you don't really need the rendezvous protocol because you will only have one namespace anyway. Thus, you could effectively replace it with a request-response protocol that just returns a list of all currently connected nodes.

The question is how are you going to establish the connection to your first node? Perhaps take a look at /dnsaddr. It allows a form of discovery by looking up multiaddresses stored in DNS records. If all your nodes support the above "peer exchange protocol" where they return all nodes they are connected to, you can pick any subset of your network and stores their addresses in the DNS record and there you've got your bootstrap mechanism.

[odesenfans]

Hello, thanks a lot for the reply! Let me clarify a few things about what I'm trying to do.

I only mentioned one type of nodes, but we actually have several. The nodes I mentioned are what we call "core" nodes, which are in charge of processing messages from application users, store them and provide consensus algorithms on top. The decisions taken by these algorithms then result in workloads/storage being deployed to secondary "worker" nodes. So we have at least 3 types of actors:

• core nodes
• secondary nodes
• application users.

Therefore, we'd have at least 2 namespaces, core and secondary. Only core nodes would be rendezvous servers, other nodes and applications would only use discovery + registration.

Regarding hosting, we do not control the full network, just a few nodes for monitoring purposes. Third-party users host the nodes themselves. When you mention censorship resistance, our strategy is basically that a few core nodes can fall without affecting the network. The few core nodes we host are typically used as bootstrap nodes so we could definitely use /dnsaddr for these, but the idea is that any core node could be a stand-in replacement.

Could you create a PR with the modification you suggest for self-registration? It seems like a useful feature regardless of this discussion.

[thomaseizinger]

When you mention censorship resistance, our strategy is basically that a few core nodes can fall without affecting the network. The few core nodes we host are typically used as bootstrap nodes so we could definitely use /dnsaddr for these, but the idea is that any core node could be a stand-in replacement.

Let's assume I can somehow cut all your hosted core nodes off from the network and force all other nodes to reboot so they loose their existing connections. Will your network succeed in re-building itself without your core nodes being available?

All core nodes being rendezvous nodes doesn't help in this situation because a node that just booted still fails at joining the network if all hardcoded bootnodes are down / inaccessible.

Please also see the "Use cases" section in the spec: https://github.com/libp2p/specs/tree/master/rendezvous#use-cases

As far as I understand the spec, this has been designed with the idea in mind that rendezvous nodes are highly available and scale horizontally whilst at the same time being agnostic about what services nodes register themselves with. In a way, it is similar to how STUN servers facilitate hole punching without knowing, what the resulting connection will be used for.

[odesenfans]

Let's assume I can somehow cut all your hosted core nodes off from the network and force all other nodes to reboot so they loose their existing connections. Will your network succeed in re-building itself without your core nodes being available?

At the moment the assumption is that different bootstrap nodes can then be selected by the community. It would require a change of configuration and a reboot of the nodes, but the situation you describe seems quite drastic. I do catch your drift that hosting rendezvous servers on the side would increase the reliability of the network.

What we currently do is that we use a pubsub topic where each node publishes a message once in a while. Each node then records peers on disk. On reboot, nodes already have the full picture of the network and don't need to re-discover everything. Meaning, bootstrap nodes are only really necessary on the first boot. What I don't like about this is that it feels like reinventing the wheel and building a tool that duplicates the functionality of rendezvous on top of gossipsub instead of using rendezvous directly. This could be done with rendezvous as well, we just need to store the namespace table on each update. We could then use the addresses + peer IDs in that table instead of the bootstrap nodes on subsequent reboots.

Overall, I'm just having trouble determining which bootstrap/discovery strategy would work the best for us.

[thomaseizinger]

Thanks for further outlining your situation.

I think I'd recommend the following setup:

• Allow users to configure a bootnode on startup / in a config file
• Default the above config to a /dnsaddr that is backed by your fleet of core nodes
• Create a request-response protocol that allows you to retrieve a list of peer ID / multiaddr pairs¹ from a node, based on its current connections
• Continue to persist useful addresses / peerIDs to disk

The combination of the above allows to:

• Specify a new bootnode in case the existing ones are gone without needing to ship a new binary, i.e. the boot node is flexible
• /dnsaddr turns a single bootnode address into a list of bootnod
• The request-response protocol allows nodes to exchange connections over time

Apart from what has already been written, another reason why I wouldn't use rendezvous here is because a rendezvous node doesn't by itself exchange records with another rendezvous node. Thus even if each core node registers itself on its own rendezvous node, there would only be a single, useless entry in there unless another node also registers itself and I doubt you want to perform 3242² registrations let alone renewing those constantly.

You could implement an extension where you share registrations via a gossipsub topic for example.

Footnotes

1. Ideally, you would be using peer records for this to avoid a malicious node from expanding the network by returning arbitrary addresses. Rendezvous uses peer records because it has no other option here. Given that your network has a consensus layer anyway, this might not be necessary. ↩

2. That is each core node registering at every other core node. ↩

[odesenfans]

Thanks a lot for the ideas, this helps a ton!