Example of nodes on different network

[fgadaleta]

What would be a best practice (or better an example) to connect to a DDS pubsub from nodes on different networks (even just different IP)?

[jhelovuo]

How far away?

Built-in Discovery should automatically connect to nodes within the same LAN (same broadcast domain).

The major problem in connecting to nodes outside the local LAN is Discovery, i.e. how the participants (nodes) find out about the existence of other participants and their Readers and Writers. One way to accomplish this is to set up some kind of DDS-level router that will forward at least Discovery traffic across networks. Some DDS vendors have software products that do exactly this. Another solution would be to bridge two distant networks into one by some VPN-type solution so that UDP multicast works across the bridge.

Yet another solution would be to forget automatic Discovery and manually configure the equivalent information to each participant. RustDDS currently has no API to supply Discovery data from the application, but that should not be very hard to implement.

When going over a public network, keep in mind that basic DDS has just about zero security in terms of e.g. cryptography or authentication.

[fgadaleta]

First of, thank you for the explanation. This makes a ton of sense.
In my "specific" scenario I have multiple nodes that communicate with each other on several topics. We can assume all such nodes are in the same network (though quite noisy and unstable because nodes can move and communicate via radio-frequency based comm.)
All such nodes also communicate with another fixed node that receives data from all the other nodes on one particular topic only.
I think I will have to implement discovery from the application directly indeed. And definitely adding it to RustDDS ;)

[nisseknudsen]

Hey @jhelovuo , I found this old issue and wondering if this was ever implemented.
I have ros2-client running on my computer and see in the log it uses my local LAN IP for UDPSender.
What I want it to do is to also discover via my overlay network's tunnel. If it used it automatically, that would be best.
If not possible, I'd also have the ability to know all relevant nodes' IP addressed (on the overlay network) and specify it directly, given an API exists.

I'm very interested in your guidance!

[jhelovuo]

Remote Participant discovery by other means than the automatic Discovery in the RTPS spec 2.5 (i.e. the SPDP and SEDP protocols) is still not implemented. If it were, then there would be an application-accessible API to interact with Discovery.

Your overlay tunnel (whatever that is) is likely not used automatically, because it is not reachable by IP multicast, or at least not advertised as such. RustDDS runs its automatic Discovery according to the RTPS spec. It opens multicast sockets to all network interfaces that are locally marked multicast-capable. Typically your LAN interface is, but local loopback is not. Then it starts running Discovery over all of the multicast interfaces.

Opening multicast sockets could cause the OS kernel do some IGMP signalling with IGMP-capable routers to route the multicast traffic further. But this depends on how your IP network is configured, and is out of scope here. Discovery traffic will go as far as your network routes it. In case there is no-one actively routing, then the scope would be only the local L2-switched LAN.

[nisseknudsen]

Thanks for the detail! It's hard for me to debug, but when deploy my ros2-client application onto my target machine (on dev machine works!), these are my logs when the app immediately crashes. Anything that springs into your eye why it would give these errors?

NFO UDPListener: new socket with address Ok(0.0.0.0:7400)

INFO UDPListener: new socket with address Ok(0.0.0.0:7410)

INFO ParticipantId 0 selected.

INFO UDPListener: new socket with address Ok(0.0.0.0:7401)

INFO UDPListener: new socket with address Ok(0.0.0.0:7411)

INFO New DomainParticipantInner: domain_id=0 participant_id=0 GUID=GUID {011299ca901a464a4c0bf88d EntityId::PARTICIPANT} security_feature_enabled=false

INFO UDPSender: Multicast sender on interface 192.168.0.23

INFO UDPSender::new() --> UDPSender { unicast_socket: UdpSocket { addr: 0.0.0.0:36567, fd: 43 }, multicast_sockets: [UdpSocket { addr: 192.168.0.23:41446, fd: 53 }] }

INFO New remote participant: SpdpDiscoveredParticipantData { updated_time: 2024-11-22T19:53:26.096016592Z, protocol_version: 2.3, vendor_id: RustDDS / Atostek, expects_inline_qos: false, participant_guid: GUID {011299ca901a464a4c0bf88d EntityId::PARTICIPANT}, metatraffic_unicast_locators: [UdpV4(192.168.0.23:7410)], metatraffic_multicast_locators: [UdpV4(239.255.0.1:7400)], default_unicast_locators: [UdpV4(192.168.0.23:7411)], default_multicast_locators: [UdpV4(239.255.0.1:7401)], available_builtin_endpoints: BuiltinEndpointSet { value: 402656319 }, lease_duration: Some(infinite), manual_liveliness_count: 0, builtin_endpoint_qos: None, entity_name: None }

INFO Remote participant GUID {011299ca901a464a4c0bf88d EntityId::PARTICIPANT} is myself, but some reflection is good.

INFO Discovery started. Participant constructed.

INFO Matched new remote reader on topic="DCPSParticipant" reader=GUID {011299ca901a464a4c0bf88d EntityId::SPDP_BUILTIN_PARTICIPANT_READER}

INFO Matched new remote reader on topic="DCPSSubscription" reader=GUID {011299ca901a464a4c0bf88d EntityId::SEDP_BUILTIN_SUBSCRIPTIONS_READER}

INFO Matched new remote reader on topic="DCPSPublication" reader=GUID {011299ca901a464a4c0bf88d EntityId::SEDP_BUILTIN_PUBLICATIONS_READER}

INFO Matched new remote reader on topic="DCPSTopic" reader=GUID {011299ca901a464a4c0bf88d EntityId::SEDP_BUILTIN_TOPIC_READER}

INFO New remote participant: SpdpDiscoveredParticipantData { updated_time: 2024-11-22T19:53:26.096016592Z, protocol_version: 2.3, vendor_id: RustDDS / Atostek, expects_inline_qos: false, participant_guid: GUID {011299ca901a464a4c0bf88d EntityId::PARTICIPANT}, metatraffic_unicast_locators: [UdpV4(192.168.0.23:7410)], metatraffic_multicast_locators: [UdpV4(239.255.0.1:7400)], default_unicast_locators: [UdpV4(192.168.0.23:7411)], default_multicast_locators: [UdpV4(239.255.0.1:7401)], available_builtin_endpoints: BuiltinEndpointSet { value: 402656319 }, lease_duration: Some(infinite), manual_liveliness_count: 0, builtin_endpoint_qos: None, entity_name: None }

INFO Remote participant GUID {011299ca901a464a4c0bf88d EntityId::PARTICIPANT} is myself, but some reflection is good.

INFO Discovery started. Participant constructed.

INFO Matched new remote reader on topic="DCPSParticipant" reader=GUID {011299ca901a464a4c0bf88d EntityId::SPDP_BUILTIN_PARTICIPANT_READER}

INFO Matched new remote reader on topic="DCPSSubscription" reader=GUID {011299ca901a464a4c0bf88d EntityId::SEDP_BUILTIN_SUBSCRIPTIONS_READER}

INFO Matched new remote reader on topic="DCPSPublication" reader=GUID {011299ca901a464a4c0bf88d EntityId::SEDP_BUILTIN_PUBLICATIONS_READER}

INFO Matched new remote reader on topic="DCPSTopic" reader=GUID {011299ca901a464a4c0bf88d EntityId::SEDP_BUILTIN_TOPIC_READER}

INFO Matched new remote reader on topic="DCPSParticipantMessage" reader=GUID {011299ca901a464a4c0bf88d EntityId::P2P_BUILTIN_PARTICIPANT_MESSAGE_READER}

INFO Matched new remote writer on topic="DCPSParticipant" writer=GUID {011299ca901a464a4c0bf88d EntityId::SPDP_BUILTIN_PARTICIPANT_WRITER}

INFO Matched new remote writer on topic="DCPSSubscription" writer=GUID {011299ca901a464a4c0bf88d EntityId::SEDP_BUILTIN_SUBSCRIPTIONS_WRITER}

INFO Matched new remote writer on topic="DCPSPublication" writer=GUID {011299ca901a464a4c0bf88d EntityId::SEDP_BUILTIN_PUBLICATIONS_WRITER}

INFO Matched new remote writer on topic="DCPSTopic" writer=GUID {011299ca901a464a4c0bf88d EntityId::SEDP_BUILTIN_TOPIC_WRITER}

INFO Matched new remote writer on topic="DCPSParticipantMessage" writer=GUID {011299ca901a464a4c0bf88d EntityId::P2P_BUILTIN_PARTICIPANT_MESSAGE_WRITER}

INFO===== RustDDS shutting down ===== .drop() DomainParticipantDisc

INFO Stopping Discovery

INFO Stopped Discovery

ERROR Event for unknown writer EntityId {[0, 0, 1] EntityKind::WRITER_NO_KEY_USER_DEFINED}

ERROR Event for unknown writer EntityId {[0, 0, 2] EntityKind::WRITER_NO_KEY_USER_DEFINED}

INFO PollEventSender.send: Broken pipe (os error 32)

INFO PollEventSender.send: Broken pipe (os error 32)

INFO PollEventSender.send: Broken pipe (os error 32)

INFO PollEventSender.send: Broken pipe (os error 32)

INFO PollEventSender.send: Broken pipe (os error 32)

INFO PollEventSender.send: Broken pipe (os error 32)

INFO PollEventSender.send: Broken pipe (os error 32)

INFO PollEventSender.send: Broken pipe (os error 32)

INFO PollEventSender.send: Broken pipe (os error 32)

INFO PollEventSender.send: Broken pipe (os error 32)

INFO PollEventSender.send: Broken pipe (os error 32)

INFO PollEventSender.send: Broken pipe (os error 32)

INFO dp_event_loop preparing to stop.

INFO PollEventSender.send: Broken pipe (os error 32)

INFO Stopping dp_event_loop

[jhelovuo]

Hi @nisseknudsen ,

The logs you have shown do not show any real error conditions. Everything before the line RustDDS shutting down are normal startup log messages. RustDDS is setting up an instance of itself (DomainParticiapnt) , which then starts a Discovery process, that starts broadcasting its presence and built-in Readers/Writers that Discovery uses. It then receives the messages from itself, and connects the local Readers/Writers to each other where appropriate (Matched new remote reader/writer).

Log line RustDDS shutting down indicates that the DomainParticipant is being shut down, i.e. its .drop() function was called. The error messages after that are not relevant, because it is just various parts getting alarmed by the other parties shutting down.

Note that all discovered network entities belong to the GUID Prefix 011299ca901a464a4c0bf88d, which is the id of the local DomainParticipant, chosen semi-randomly at startup. So this DDS instance is only talking to itself, presumably because no others are present on the network.

If your application was crashing, it is not visible here. If your Rust application is really crashing, then I would expect it to panic instead of just e.g. segfaulting. That should produce at least an output of the crash point, if not a full stack backtrace.