Allow two yarplogger in the network

[S-Dafarra]

We recently tested a network configuration where there are two subnets communicating with each other while being in the same YARP network. These two networks are in two different rooms. When testing, we need to launch several applications from either room, but unfortunately the logger can be in a single room because of https://github.com/robotology/yarp/blob/b418c1b37ab7e317369e64f2840bfad3881ad7f5/src/libYARP_logger/src/yarp/logger/YarpLogger.cpp#L505-L509.
I suppose that after robotology/yarp#2434 it might be possible to run two different loggers. I guess that the repeater should open the port /yarlogger (usually used by yarprun with the --log option https://github.com/robotology/yarp/blob/b418c1b37ab7e317369e64f2840bfad3881ad7f5/src/libYARP_os/src/yarp/os/impl/LogForwarder.cpp#L44), while the two instances of the yarplogger should start with a different name (https://github.com/robotology/yarp/blob/b418c1b37ab7e317369e64f2840bfad3881ad7f5/src/yarplogger/mainwindow.cpp#L319).

Does this make any sense? Would it be possible to have this done automatically?

cc @traversaro @pattacini @randaz81 @drdanz

[traversaro]

Hi @S-Dafarra, sorry, I had missed the first time you opened this, sorry! If I understood correctly, this problem is just related due to the fact that the yarplogger is both the central collector of log messages, and the GUI to visualize such messages? Instead the subnet here do not play a role, as you would have the same problem if you wanted to run two instances of the logger in the same machine or in two machines in a normal network?

Having said that, I think the workaround that you propose should work fine. I guess it could be of interested of other persons that are working on logging at different levels (@vtikha @marcoaccame @Nicogene @triccyx).

[S-Dafarra]

If I understood correctly, this problem is just related due to the fact that the yarplogger is both the central collector of log messages, and the GUI to visualize such messages?

Yes, that is correct.

Instead the subnet here do not play a role, as you would have the same problem if you wanted to run two instances of the logger in the same machine or in two machines in a normal network?

That's true. Although it was not really problematic when all the PC of the setup were "close". Having the logger running on a single PC was not much of a deal. Being in two physically separated rooms instead, makes it a bit not practical.

[S-Dafarra]

Another possibility is also to split the messages to two different loggers. I checked the yarprun code, and actually after log https://github.com/robotology/yarp/blob/b418c1b37ab7e317369e64f2840bfad3881ad7f5/src/libYARP_run/src/yarp/run/Run.cpp#L156
it seems to be possible to select a port name different from /yarplogger to stream the output. This would allow redirecting the output of different yarprun to different loggers.

On the other hand, if the yarprun is started from the yarpmanager, this possibility is not available. From the cluster-config file, the log variable is not considered https://github.com/robotology/yarp/blob/b418c1b37a/src/libYARP_manager/src/yarp/manager/xmlclusterloader.cpp#L37-L145
and always set to true: https://github.com/robotology/yarp/blob/b418c1b37a/src/libYARP_manager/src/yarp/manager/xmlclusterloader.h#L29

[traversaro]

Interesting, thanks! I guess that if that turns the best options to run two yarplogger, probably it could make sense to support that option in the yarpmanager GUI, what do you think @Nicogene @drdanz @randaz81 ?

[drdanz]

Sorry @S-Dafarra, I missed this discussion too...

In my opinion, the best way to implement this, could be to remove all the magic of the /yarplogger port, and make it a ROS-like "topic", with the yarplogger and yarplogger-console executables that acts as a subscriber, and yarprun, and all the executables (when YARP_FORWARD_LOG_ENABLE is set) that act as publishers.
The main issue with this approach is that, if 2 loggers are listening, the generated network traffic is double

Another option would be, as you suggest, to choose a different name for the /yarplogger (via environment variable, or program argument)
This approach has a different issue... IIRC, when you start the logger, it gets the list of the port on the network, and connect to all the ports that start with /log/. This allows you to start the logger while the programs are already running, but unfortunately it means that the /yarplogger port is not the only port that should be configurable...

[S-Dafarra]

In my opinion, the best way to implement this, could be to remove all the magic of the /yarplogger port, and make it a ROS-like "topic", with the yarplogger and yarplogger-console executables that acts as a subscriber, and yarprun, and all the executables (when YARP_FORWARD_LOG_ENABLE is set) that act as publishers.
The main issue with this approach is that, if 2 loggers are listening, the generated network traffic is double

I agree. The double network traffic would be there also with the workaround with yarp repeat indeed. All the logs will have to reach yarp repeat and from this go to all the instances of yarplogger.

This approach has a different issue... IIRC, when you start the logger, it gets the list of the port on the network, and connect to all the ports that start with /log/. This allows you to start the logger while the programs are already running, but unfortunately it means that the /yarplogger port is not the only port that should be configurable...

But is this a problem? Suppose that we start a yarplogger with the default name, and one with a different name, let's say yarplogger2. When starting a new module, the log will automatically appear in yarplogger. If then I start yarplogger2, it automatically connects to all the /log/ ports (clearly, this is not ideal for the traffic point of view), and it will also start receiving the messages, right? If a new module starts, its entry will not appear in yarplogger2 (since the automatic connection is only towards the /yarplogger port), but what happens if I simply press the refresh button of yarplogger2? Will that entry appear?

[drdanz]

The main issue is, in my opinion reproducibility... The system has a different behaviour (different connections), depending on the order of how the components are started.

Anyway, assuming that we can forget about that, this is how it would work (exe# can either be an executable started by yarprun, or using YARP_FORWARD_LOG_ENABLE).

• exe1 starts with the default settings, therefore opens a port /log/exe1/xxx and tries to connect to /yarplogger (connection fails since the port is not available)
• exe2 starts with the logger port = /ylog2, therefore opens a port /log/exe2/xxx and tries to connect to `/ylog2`` (connection fails since the port is not available)
• yarplogger1 starts with the default settings, opens the port /yarplogger, notices /log/exe1/xxx and /log/exe2/xxx and connects to both (/log/exe2/xxx to /yarplogger connection is estabilished, while it should not be).
• yarplogger2 starts with the logger port = /ylog2, opens the port /ylog2, notices /log/exe1/xxx and /log/exe2/xxx and connects to both (/log/exe1/xxx to /ylog2 connection is estabilished, while it should not be).
• exe3 starts with the default settings, therefore opens a port /log/exe3/xxx and tries to connect to /yarplogger (connection successful)
• exe4 starts with the logger port = /ylog2, therefore opens a port /log/exe4/xxx and tries to connect to `/ylog2`` (connection successful)

At this point the executables that were started before the loggers (exe1 and exe2) are publishing to both the loggers and therefore generating double network traffic, while those who were started after (exe3 and exe4) are pubblishing to the right one.

[S-Dafarra]

That's clear, thanks!

• yarplogger1 starts with the default settings, opens the port /yarplogger, notices /log/exe1/xxx and /log/exe2/xxx and connects to both (/log/exe2/xxx to /yarplogger connection is estabilished, while it should not be).
• yarplogger2 starts with the logger port = /ylog2, opens the port /ylog2, notices /log/exe1/xxx and /log/exe2/xxx and connects to both (/log/exe1/xxx to /ylog2 connection is estabilished, while it should not be).

I think that these two cases are not much of a big deal, as long as the two loggers start as the first modules.

And in case we do want to see the same messages in two locations (accepting the increased network load), we can simply restart the logger with the non-standard configuration. Am I correct?

Also, is it really necessary to restart the logger, or we could simply hit the refresh button?

[drdanz]

And in case we do want to see the same messages in two locations (accepting the increased network load), we can simply restart the logger with the non-standard configuration. Am I correct?

I think you might be able to do this already, using YARP_PORT_PREFIX or YARP_RENAME.
But this will not get anything that was sent before the start, and will not receive any other connection

Also, is it really necessary to restart the logger, or we could simply hit the refresh button?

I think it is necessary to restart the logger, but I might be wrong.

[S-Dafarra]

I see, thanks! The more we discuss though, I get somehow convinced that they are basically unpractical workarounds.
We would have to restart the second logger every time manually, fast enough to get the initial failure messages, but not too fast in order to connect to all the /log ports 😅 .

Eventually, it might be ok to redirect all the log coming from the robot head to a dedicated logger, but in general, having two loggers seems problematic.

[S-Dafarra]

In any case, just to be clear about the actual needs, the main usage of the yarplogger for us is to understand why a module did not start when using the yarpmanager. In the 99% of the case, we only save to file the log from the robot ```yarprobotinterface``. For all the other modules we only need to check the output quickly.

We are trying to exploit more and more yarpmanager. We run dozens of applications at the same time on 4-5 different laptops with a single click. That's very powerful. The problem is when some module refuses to start. If that happens, we need to go to the yarplogger, but that will most probably have several dozens of entries (from the previous runs), so searching for the correct entry becomes a needle in the haystack problem.
So, in the end I guess that our main need would be to quickly check why a module did not start from the yarpmanager

Eventually, I can also start another discussion for this.