How to connect two Fast-DDS Discovery Servers running on separate PCs?

[royito55]

Hi, I have a setup with 2 create3 robots (create3 base + rasperry pi, using discovery server and communication through usb0 cable), and I want them to be able to see them both with ros2 topic list.

They are both working fine on their own, but when I try to see both robots from a third PC, I can only see one at a time, and including both ips in the super client configuration file does not seem to work.

Here is my setup:

robot 1:

ROS_DISCOVERY_SERVER="127.0.0.1:11811;192.168.1.robot_2:1182"
FASTRTPS_DEFAULT_PROFILES_FILE=(default super_client_configuration_file.xml @ 127.0.0.1)

robot 2:

ROS_DISCOVERY_SERVER="127.0.0.1:11812;192.168.1.robot_1:11811"
FASTRTPS_DEFAULT_PROFILES_FILE=(default super_client_configuration_file.xml @ 127.0.0.1)

PC:

ROS_DISCOVERY_SERVER="192.168.1.robot_1:11811;192.168.1.robot_2:11812"
FASTRTPS_DEFAULT_PROFILES_FILE=
<?xml version="1.0" encoding="UTF-8" ?>
<dds>
    <profiles xmlns="http://www.eprosima.com/XMLSchemas/fastRTPS_Profiles">
        <participant profile_name="super_client_profile" is_default_profile="true">
            <rtps>
                <builtin>
                    <discovery_config>
                        <discoveryProtocol>SUPER_CLIENT</discoveryProtocol>
                        <discoveryServersList>
                            <RemoteServer prefix="44.53.00.5f.45.50.52.4f.53.49.4d.41">
                                <metatrafficUnicastLocatorList>
                                    <locator>
                                        <udpv4>
                                            <address>192.168.1.72</address>
                                            <port>11811</port>
                                        </udpv4>
                                    </locator>
                                </metatrafficUnicastLocatorList>
                            </RemoteServer>
                            <RemoteServer prefix="44.53.00.5f.45.50.52.4f.53.49.4d.41">
                                <metatrafficUnicastLocatorList>
                                    <locator>
                                        <udpv4>
                                            <address>192.168.1.190</address>
                                            <port>11812</port>
                                        </udpv4>
                                    </locator>
                                </metatrafficUnicastLocatorList>
                            </RemoteServer>
                        </discoveryServersList>
                    </discovery_config>
                </builtin>
            </rtps>
        </participant>
    </profiles>
</dds>

THE ABOVE XML CONFIG DOES NOT WORK. It only works if I comment out one <RemoteServer>. Afterwards, I can see the other one fine.

Due to the nature of the robots, I have to run a discovery server in each robot, so something like running the server in the PC and pointing the ROS_DISCOVERY_SERVER to the PCs IP is not an option.

Any ideas?

[JesusPoderoso]

Hi @royito55
If I've understood your deployment, you have configured:

• Robot 1 as super client, with discovery server 0 as PC (localhost) and discovery server 1 as Robot 2
• Robot 2 as super client, with discovery server 0 as PC (localhost) and discovery server 1 as Robot 1
• PC as super client, with discovery server 0 as 192.168.1.72 and discovery server 0 as 192.168.1.190

It seems that you are configuring all entites as super client instead of discovery servers, so make sure that tag <discoveryProtocol>SUPER_CLIENT</discoveryProtocol> is replaced with the tag <discoveryProtocol>SERVER</discoveryProtocol> in the entities you want to deploy as discovery server.

In those you perform the change, make sure to also introduce the GUID prefix of the server and the metatraffic unicast locator to set at which address and port is the server listening for clients.
Finally, you should launch the discovery server too.

I recommend you to read and follow this instructions.

[royito55]

Hi, thank you for your answer. Let me try to explain better:

• Robot 1 is running a discovery server (fastdds discovery -i 0). I also added the super client config file variable in order to see the robot 1 own topic. I also specified two servers with ROS_DISCOVERY_SERVER: one running within itself (localhost); and the other running in Robot 2 (192.168.1.190)

• Robot 2 is running a discovery server (fastdds discovery -i 0 -p 11812, pretty sure I don't need a different port but just in case). I also added the super client config file variable in order to see the robot 2 own topic. I also specified two servers with ROS_DISCOVERY_SERVER: one running within itself (localhost); and the other running in Robot 1 (192.168.1.72)

• PC I want to set as a super client. It is NOT running a discovery server, but I specified the ones running in Robot 1 and Robot 2 with the variable ROS_DISCOVERY_SERVER="192.168.1.72:11811;192.168.1.190:11812". What I'm looking for is being able to see both robots topics in the PC, and eventually run nodes that communicate with robot1, robot2, robotN...

Should I be running a discovery server in the PC? I'm not doing so currently.

I will try changing that tag from SUPER_CLIENT to SERVER. But, won't I lose the super client capabilities in the PC this way?

Hope I'm explaining myself.

[royito55]

I just realized you meant changing the tags in the PCs running the servers, my bad.

How would I go about introducing the GUID prefix of the server and the metatraffic unicast locator to set at which address and port is the server listening for clients? Does something like this (In a robot running a server) make sense??

<?xml version="1.0" encoding="UTF-8" ?>
<dds>
    <profiles xmlns="http://www.eprosima.com/XMLSchemas/fastRTPS_Profiles">
        <participant profile_name="super_client_profile" is_default_profile="true">
            <rtps>
                <builtin>
                    <discovery_config>
                        <discoveryProtocol>SERVER</discoveryProtocol>
                        <discoveryServersList>
                            <RemoteServer prefix="44.53.00.5f.45.50.52.4f.53.49.4d.41">
                                <metatrafficUnicastLocatorList>
                                    <locator>
                                        <udpv4>
                                            <address>192.168.1.83(SUPER CLIENT PC)</address>
                                            <port>11811</port>
                                        </udpv4>
                                    </locator>
                                </metatrafficUnicastLocatorList>
                            </RemoteServer>
                        </discoveryServersList>
                    </discovery_config>
                </builtin>
            </rtps>
        </participant>
    </profiles>
</dds>

[EduPonz]

Hi @royito55 ,

The problem is that you are running both servers with the same ID. Try running:

Robot 1

On one terminal, run the server with:

fastdds discovery -i 0

On the other terminals where you are running nodes, remember to export:

export ROS_DISCOVERY_SERVER="127.0.0.1:11811" 

This configures your nodes as Clients to the Discovery Server with ID 0

Robot 2

On one terminal, run the server with:

fastdds discovery -i 1

On the other terminals where you are running nodes, remember to export:

ROS_DISCOVERY_SERVER=";127.0.0.1:11811" 

This configures your nodes as Clients to the Discovery Server with ID 1 (mind the ; just in from of the localhost address)

PC

To see all the topics on the PC, you'll need a Super Client that connects to both Servers (ID 0 and ID 1). This is done by applying the following XML:

<?xml version="1.0" encoding="UTF-8" ?>
<dds>
    <profiles xmlns="http://www.eprosima.com/XMLSchemas/fastRTPS_Profiles">
        <participant profile_name="super_client_profile" is_default_profile="true">
            <rtps>
                <builtin>
                    <discovery_config>
                        <discoveryProtocol>SUPER_CLIENT</discoveryProtocol>
                        <discoveryServersList>
                            <RemoteServer prefix="44.53.00.5f.45.50.52.4f.53.49.4d.41">
                                <metatrafficUnicastLocatorList>
                                    <locator>
                                        <udpv4>
                                            <address>192.168.1.72</address>
                                            <port>11811</port>
                                        </udpv4>
                                    </locator>
                                </metatrafficUnicastLocatorList>
                            </RemoteServer>
                            <!--
                                Mind that the prefix in this case is not the same as for the other server
                                You can get the prefix from the terminal in which you run the server
                            -->
                            <RemoteServer prefix="44.53.01.5f.45.50.52.4f.53.49.4d.41">
                                <metatrafficUnicastLocatorList>
                                    <locator>
                                        <udpv4>
                                            <address>192.168.1.190</address>
                                            <port>11811</port>
                                        </udpv4>
                                    </locator>
                                </metatrafficUnicastLocatorList>
                            </RemoteServer>
                        </discoveryServersList>
                    </discovery_config>
                </builtin>
            </rtps>
        </participant>
    </profiles>
</dds>

Then, on the terminal where you want to run the ros2 CLI, do:

export FASTRTPS_DEFAULT_PROFILES_FILE=<path_to_the_xml>
ros2 daemon stop
ros2 daemon start
ros2 topic list

[royito55]

I think there is an issue when changing the ID of the server. It seems like I can't discover it in any way. I'm making sure to change the GUID to reflect the prefix change but still no luck. As soon as I switch back to ID 0, things work. Here is what is happening:

[royito55]

I haven't been able to launch a server with an ID different than zero that allows me to see the robot topics, in any computer.

[royito55]

I think I figured out that the Super Client is what is not working correctly.

I tried doing a talker from the PC, and both robots heard it.

Conversely, I did a listener in the PC, and it heard both robot talkers:

So maybe something is off with Super Clients and Server IDs?

[EduPonz]

Hi again,

I have prepared a small docker compose example to illustrate how to configure this (remote_pc_ds.zip). Important points:

1. robot 1 runs a server with ID 0 and a talker a listener demo pointing to the ID 0 server using localhost address
2. robot 2 runs a server with ID 1 and a content_filtering_publisher a content_filtering_subcriber demo pointing to the ID 1 server using localhost address
3. PC runs the ROS 2 daemon a Super Client with two remote servers identified by:
   1. Their corresponding prefixes
   2. The addresses of the container for robot 1 and robot 2

You'll see that the pc container output shows both the /chatter and /temperature topics without problems. I thinks there may be something wrong with your superclient xml, probably the IP addresses of the remote servers

[royito55]

Well, I triple checked the XML, and the IP addresses are correct. I still have the same problem.

Maybe the fact that you use a different IP class in your example than what I have available is the problem? I see you are using subnet 172.33.0.0/24 and addresses 172.33.0.2, 172.33.0.3

Sorry to not be well versed in IPs, but I'm just using the most common one, just by looking at the wlan0 interface with ifconfig so they are 192.168.1.72 and 192.168.1.190. So they are definitely correct in the superclient xml:

<?xml version="1.0" encoding="UTF-8" ?>
<dds>
    <profiles xmlns="http://www.eprosima.com/XMLSchemas/fastRTPS_Profiles">
        <participant profile_name="super_client_profile" is_default_profile="true">
            <rtps>
                <builtin>
                    <discovery_config>
                        <discoveryProtocol>SUPER_CLIENT</discoveryProtocol>
                        <discoveryServersList>
                            <RemoteServer prefix="44.53.00.5f.45.50.52.4f.53.49.4d.41">
                                <metatrafficUnicastLocatorList>
                                    <locator>
                                        <udpv4>
                                            <address>192.168.1.190</address>
                                            <port>11811</port>
                                        </udpv4>
                                    </locator>
                                </metatrafficUnicastLocatorList>
                            </RemoteServer>
                            <RemoteServer prefix="44.53.01.5f.45.50.52.4f.53.49.4d.41">
                                <metatrafficUnicastLocatorList>
                                    <locator>
                                        <udpv4>
                                            <address>192.168.1.72</address>
                                            <port>11811</port>
                                        </udpv4>
                                    </locator>
                                </metatrafficUnicastLocatorList>
                            </RemoteServer>
                        </discoveryServersList>
                    </discovery_config>
                </builtin>
            </rtps>
        </participant>
    </profiles>
</dds>

[royito55]

As a followup, I possibly identified the issue and I want to make sure it makes sense.

For the robot running the Discovery Server with ID = 1, you then specify to set ROS_DISCOVERY_SERVER=";127.0.0.1:11811" with the added ; when launching nodes in that robot.

I'm not planning on launching nodes in the robot, but there are already some (inaccessible) robot driver nodes offered by the iRobot base (which publish the topics I'm interested in). The way iRobot has developed the Fast-DDS server config is through this web server:

It sounds like the "Address and port of Fast DDS discovery server:" sets the variable ROS_DISCOVERY_SERVER. Now, following the logic from above, if I understand correctly, I should then change "192.168.186.3:11811" (a wired usb0 address from iRobot base to RPi) to ";192.168.186.3:11811".

Does that make sense? What is the ; for?

If that's the case, then I'm kind of screwed because the iRobot doesn't let you add the ; before the address to set ROS_DISCOVERY_SERVER in that setup page (iRobot should fix that!)

[EduPonz]

Hi again,

There are several things to comment here, I hope to clarify all of them.

First, I think there's been a mixup of different concepts:

1. Although any DDS DomainParticipant can be configured to act as Discovery Server, the normal operation is to instantiate one using the fastdds discovery CLI (see docs).

2. The environment variable ROS_DISCOVERY_SERVER is NOT used to instantiate Discovery Servers. It is used to connect the DomainParticipant to a set of Discovery Servers (see docs). As the documentation states, this env var is designed to be used in combination with fastdds discovery CLI. The env var is a list of ; separated UDPv4 locators (address-port pairs), each one corresponding to one Discovery Server ID, i.e. the first entry corresponds to the Server with ID 0, the second with the Server with ID 1 and so on.

3. If this env var is set when instantiating a Server, that Server will try to connect to the set of Servers specified in the env var. For example:

   export ROS_DISCOVERY_SERVER=";;192.168.1.54:11811"
   fastdds discovery -i 1

   will instantiate a Server with ID 1 that connect to a Server with ID 2 which is listening on 192.168.1.54:11811

4. When Servers are inter-connected, they shared the information about all their clients

5. When using the Discovery Server, the Domain ID does not matter, the Servers act as different domains. In that sense, connecting 2 servers results in fussing the two "domains".

Going back to your example, please verify that that you are running fastdds discovery -i 0 in the machine with IP 192.168.1.190 and fastdds discovery -i 1 in the machine with IP 192.168.1.72.

Now, coming to the Create3, the one Server configuration that I cannot see is the ID; from that I'd assume that they are simply using ID 0. This has several impications:

1. To connect to the Create3 Server, you'd need to set ROS_DISCOVERY_SERVER=192.168.186.3:11811
2. You cannot connect to two Create3 Servers at the same time, as both of them have the same ID (pinging @alsora for clarification on this)