Participants discovered by Discovery Server, but Writers and Readers do not match in WAN [DDS over WAN], Some questions

[ploe24]

Hello,

right now we have our own DS with a public IP.
There are two LANs containing X participants each (Subscribers and Publishers).
All participants connecting to the DS via its public IP and port. The DS recognizes the participants, but the DataReaders and -Writers do not match (even they share the same topic and data types) and therefore no data is transmitted. So besides the successful discovery process nothing seems to happen.

After doing some research we stumbled upon "eProsima DDS Router" to connect and direct data from multiple LANs across the Internet/over WAN.
Is there no other way?
The section 6.3.3. WAN or Internet Communication over TCPv4 describes a communication over WAN. But is it a point-2-point connection of one publisher and one subscriber or can you use a DS in this case and use X subscribers in combination with Y publishers?!
The example mentions that the publisher acts as TCP "server" to start the process while the "client" (subscriber) has the address of the server in its "Initial peer" list. How does it work when having multiple subscribers and publishers?
How would a connection like this look like when having a public IP DS "in the middle"?

Very much appreaciated if somebody could enlighten me, how to configure our DDS setup or at least point in the right direction where to continue and what could have went wrong.

One more question: what ports have to be forwarded in our public router (in front of the DS)? Only the port, where the server is listening on (something like 16166) or do we have to care about the RTPS ports (7400, 7401 and 7402)?

Thanks in advance

If more information is needed please let me know.
The code for the server, pubs and subs is mainly from the eProsima Discovery Server example with some little modifications and addresses adapted to our network config.

[JesusPoderoso]

Hi @ploe24, thanks for using Fast DDS.

The section 6.3.3. WAN or Internet Communication over TCPv4 describes a communication over WAN. But is it a point-2-point connection of one publisher and one subscriber or can you use a DS in this case and use X subscribers in combination with Y publishers?!

you are right, that section does not describe a TCP connection over WAN using Discovery Server as discovery mechanism.
Can you check if configuring the discovery server using this XML profile and the clients using this other XML profile suits your requirements? Let us know.

One more question: what ports have to be forwarded in our public router (in front of the DS)? Only the port, where the server is listing on (something like 16166) or do we have to care about the RTPS ports (7400, 7401 and 7402)

You should just forward the physical port (your first 16166 option), RTPS ports are used as virtual/logical ports in TCP. See this DDS Router section for transversal NAT configuration if you need more information.

[ploe24]

@JesusPoderoso Thank you for your fast reply.
We will try and test few things in the next days. After that I will let you know if it worked out.

Just to be sure:
The XML tags <port> and <physical_port> as shown in your linked example by VulcanNexus are equal to

IPLocator::setIPv4(locator, 111, 111, 111, 111);
IPLocator::setPhysicalPort(locator, 5555); // this port has to be forwarded in NAT?
IPLocator::setLogicalPort(locator, 12345);

right?

If so, we might found a potential bug in FastDDS 2.11.3

Since we are using parts of the DS example, the server reports:

Server-Participant "MyDDS_Server" created with GUID XX.XX.XX.XX.XX.XX.XX.XX.XX.XX.XX.XX | 0.0.1.c1 listening in address <TCPv4:[IP.IP.IP.IP]:1059471142>

where IP is our public IP, but 1059471142 is not the port we've set. It's actually not a valid port number anyway.
Found out that the number changes according to the actual port that was set (here 16166).
The output as above is generated by a std::cout << locator as in the Discovery Server example.
Could there be a bug in the << operator of Locator class?!
The "high" and strange value is always

Actual port number = printed number / (64 * 1024)

Doesn't matter if you set a port 10, 12, 10.000 or whatever. The displayed number in the console is always around 64*1024 times higher.

Will check if upgrading to latest FastDDS 2.13.3 will fix this.

Thanks again.

[EduPonz]

Hi @ploe24,

The port within a Locator is a 16-byte array, where the 8 MSB correspond to the physical port (set to zero unless the application sets it as you're doing), and the 8 LSB correspond to the logical port. That is the reason why you see these large numbers when using the << operator. In fact, in Fast DDS v2.11.3, it the locator kind is set to TCPv4 or TCPv6, the printed locator should show as TCPv4:[IP.IP.IP.IP]:<physical_port>-<logical_port> (see here).

[ploe24]

In fact, in Fast DDS v2.11.3, it the locator kind is set to TCPv4 or TCPv6, the printed locator should show as TCPv4:[IP.IP.IP.IP]:<physical_port>-<logical_port> (see here).

Yes, I'm referring to the DS example from 2.11.3, esp. the output starting at this line here.
If retrieved with IPLocator::getLogicalPort(locator) the port is correct, but std::cout << locator as used as in the example prints something completely off.
As long as it's just a console output mistake and the server is not trying to listen on a port like 1059471142, it's fine :)
Will update to latest FastDDS (2.13.3) next week and see if everything works (including expected output) then 👍

[ploe24]

@EduPonz @JesusPoderoso
Thank you guys for your helpful input so far.

Yesterday we installed FastDDS 2.13.2, the latest pre-compiled build for Win x64 available, from the eProsima download section, but there seems to be something wrong with the lib.
We got the error below and no project worked, not even the examples (same LNK2019 error).
(Btw: doesn't matter if we use the debug or release mode library)

Unresolved external symbol "void __cdecl eprosima::set_name_to_current_thread_impl<char const *>(char const *,char const *)" in librtpsd-2.13.lib

So we ended up with FastDDS 2.12.1, which we are using now and it works fine.

Anyway, coming back to my initial question :)

Our setup is like the third case in the graphics here and we re-created the XML config as suggested by @JesusPoderoso for our discovery server and our pubs/subs using the API functions. The DS sits in our DMZ (all ports forwarded) and the participant discovery works as before. But still no data transfer and the Writers and Readers are not matching :(

For my further understanding, is it mandatory to use FastDDS-Router (this software)? Or is a communication over WAN/Internet also possible without it?
It seems that Vulcanexus is also using some software router running in a docker container, as seen here.
Do we also need it?

TIA :-)

Edit:

In 6.3.3. WAN or Internet Communication over TCPv4 it looks like there is no "FastDDS Router" involved, but also no public discovery server. So the "Initial Peers" setup won't work for us, right? Because this is what we tried using the config as suggested

you are right, that section does not describe a TCP connection over WAN using Discovery Server as discovery mechanism.
Can you check if configuring the discovery server using this XML profile and the clients using this other XML profile suits your requirements? Let us know.

[JesusPoderoso]

Hi @ploe24, thanks for your report.

Yesterday we installed FastDDS 2.13.2, the latest pre-compiled build for Win x64 available, from the eProsima download section, but there seems to be something wrong with the lib.
We got the error below and no project worked, not even the examples (same LNK2019 error).

Sorry for the inconvenience, we've double-checked the Windows Installer for v2.13.2 and it works in our side.
Which version of Visual Studio are you using?
Also, there is no such a librtpsd-2.13.lib installed, the most similar lib is libfastrtpsd-2.13.lib, please, check that so we can further investigate.

...But still no data transfer and the Writers and Readers are not matching :(

The key point is that you need to forward ports in each participant too, not just in the discovery server side.

"TCP is a connection oriented transport, so the DomainParticipant must establish a TCP connection to the remote peer before sending data messages. Therefore, one of the communicating DomainParticipants (the one acting as server) must open a TCP port listening for incoming connections, and the other one (the one acting as client) must connect to this port" (see TCP transport section of Fast DDS documentation).

That means that no matter how the participants (and endpoints) have discovered each other (multicast, initial peers, or using a discovery server), one of them must act as a server of the TCP communicatión, who actively establishes the connection. And the other as a TCP client.
As long as who acts as TCP server or TCP client may change, both participants need their physical ports to be forwarded too.

For my further understanding, is it mandatory to use FastDDS-Router (this software)? Or is a communication over WAN/Internet also possible without it?

DDS router, among other features, makes easier to configure complex deployments over WAN, but it can be configured directly through Fast DDS.

In 6.3.3. WAN or Internet Communication over TCPv4 it looks like there is no "FastDDS Router" involved, but also no public discovery server. So the "Initial Peers" setup won't work for us, right? Because this is what we tried using the config as suggested

Discovery server is really useful while deploying large number of participants, so you configure a single (server) locator instead of N initial peers locators (and that configuration should be performed in each participant, so the number increases significantly while including participants in the deployment).
For simple deployments as the one referred (just two participants), initial peers would suite you too (but you would need to forward ports anyways).

[ploe24]

Which version of Visual Studio are you using?

MS VS 2019

Also, there is no such a librtpsd-2.13.lib installed, the most similar lib is libfastrtpsd-2.13.lib, please, check that so we can further investigate.

That was a typo on my side ;-) I reformatted the error msg and mis-spelled the filename.
Yes, of course it's libfastrtpsd-2.13.lib as well as libfastrtps-2.13.lib.
The whole package that is installed from the eProsima_Fast-DDS-2.13.2-Windows.exe installer. Configured one HelloWorldExample with CMake 3.26 and got this message in Visual Studio.
Whereas the built lib (2.12.1) that comes with the eProsima_Fast-DDS-2.12.1-Windows.exe installer works.

That means that no matter how the participants (and endpoints) have discovered each other (multicast, initial peers, or using a discovery server), one of them must act as a server of the TCP communicatión, who actively establishes the connection. And the other as a TCP client.
As long as who acts as TCP server or TCP client may change, both participants need their physical ports to be forwarded too.

Ah, yes... makes sense now :)

Now we managed to make the HelloWorldTCP example work (with static initial peer list and a forwarding of both computers running the participants).
Next step will be adapting it to our DS project...

DDS router, among other features, makes easier to configure complex deployments over WAN, but it can be configured directly through Fast DDS.

So, for example, instead of forwarding ALL participants which the DS could discover, one by one, you can set your FastDDS Router's address and then it takes care of the connected LAN(s) and directs the incoming data according to the configured route to each participant?!

Thank you very much, that helped a lot.

[JesusPoderoso]

So, for example, instead of forwarding ALL participants which the DS could discover, one by one, you can set your FastDDS Router's address and then it takes care of the connected LAN(s) and directs the incoming data according to the configured route to each participant?!

By using the DDS router as a repeater, participant configuration gets significantly reduced. Participants would discover each other simply by using multicast, and you would only need to configure the DDS Router's WAN participants with the DDS Router Repeater locators (as initial peers).
That configuration would require forwarding a single port exclusively on the DDS Router Repeater, but the only drawback is that all the traffic would pass through your DMZ, which may not be desirable.

[ploe24]

@EduPonz @JesusPoderoso

Now everything works as expected, even with DS :)