What is the reconnection mechanism of opcua?

[beyondyinjl2]

Use the subscription mode, after the opcua service is disconnected, reconnect
@OverRide
public void onSubscriptionTransferFailed(UaSubscription subscription, StatusCode statusCode)
Clear the subscriber, re-subscribe, this is no problem

However, after the opcua service is disconnected for a long time (one day), it will not reconnect. What is the reconnection mechanism of opcua?

[kevinherron]

An overview of the reconnect sequence is described in Part 4, Section 6.7.

Milo's implementation is mostly handled in SessionFsm/SessionFsmFactory and another FSM that manages the underlying TCP connection.

You can troubleshoot these by turning loggers org.eclipse.milo.opcua.sdk.client.SessionFsm and org.eclipse.milo.opcua.stack.client.ChannelFsm to DEBUG or TRACE level.

I am not aware of any issues with reconnection failing within Milo (though I have seen some servers that have issues with parts of the reconnect sequence like subscription transfers causing the connection to close). The few times there has been an issue on the client side it was somebody who was doing a blocking action that required the session inside one of the callbacks or something like that.

[ronytesler]

How do you control the logs' level?

[kevinherron]

Milo does not provide a logging implementation, it logs via the SLF4J logging facade API.

How you set a logger's level is dependent on the logging implementation you use. Often it is via a configuration file, and if your product chooses to do so it can also expose an interface for setting them dynamically at runtime.

[ronytesler]

@kevinherron Is there any way to make the reconnection time shorter? We found about the 'RetrialDelay' and 'ReconnectionPeriod' properties, but I don't think they exist in milo.
After our server's network cable is disconnected, and then connected again, it takes a lot of time until the client reconnects to it. In that time, other clients can initiate new connections to the server successfully.
How can we debug why it happens?

[kevinherron]

It's not configurable right now - it uses an exponential back-off between connection attempts, starting with an immediate attempt, and then backing off until a max of delay of 16 seconds.

How long are you waiting before plugging the cable in? Are you sure you're not also measuring how long it takes to even detect it's disconnected? When you unplug an ethernet cable you don't get a "clean" break, and you won't even know you're disconnected until the OS finally figures it out or Milo's keep-alive mechanism decides it's time to reconnect.

[innaSigal]

Hello @kevinherron,
In our system the milo OPCUA client usually behaves as expected.
But there are scenarios in which after few hours of data stream we see that the client starts to throw a warning "no pending request with requestId....".
From inspecting the Wire Shark we do see that there is a Publish request sent and it takes for about 30 secs for the Publish Response to arrive.
We also can see in Wire Shark that between this publish request and the publish response the client sends health Read Requests to the server which are responded within few miliseconds with status 0.
But for some reason this doesn't trigger the onKeepAliveNotification method in our client Java code. Indeed this method is triggered for the last time just before the message with a non existing request id arrives. This behavior is consistent across our systems.
We have 2 concerns here:

1. Why the keep alive notification is not pushed to the upper levels?
2. If this is as a result of the fact that "something wrong" happened with the server responses (the req ids which are probably timed out) why the milo client doesn't initiate reconnection?
   Thank you!

[kevinherron]

There's actually two keep alive mechanisms, one at the Subscription level, which is what the onKeepAliveNotification method is for, and one at the "application layer" - the periodic read for the server state issued by the client.

Subscription keep alive notifications are sent by the server after a certain amount of time has elapsed, based on subscription settings, without any data changes. You can read about this in OPC UA Part 4.

The additional keep alive implemented by the Milo client is there to more quickly detect when a server has stopped responding for some reason, e.g. it crashed, or a VPN connection went down, ethernet unplugged as above, or whatever else.

The "no pending request with requestId...." warnings you see indicate there may be something wrong with the server, but it's not something wrong enough to initiate a reconnect. These can basically happen any time a server eventually sends a response to something but it's after the client timeout value. When it happens with PublishResponses it usually means the server has a bug in its subscription implementation.

[innaSigal]

Hi @kevinherron,
Thank you for this very quick response.
Supposing we cannot wait for the server to resolve the bug, what will be the best solution from client side?
Do we have any access/callback to the trigger when such scenario happens -
there are responses from server side in general (the server is up and responding) but there are timeouts on the OPCUA data request.
We could check the keepalive last status response and to reset the connection based on it, but then we will also intervene the OPCUA mechanist of reconnections in case there is no response at all from the server (we can, but maybe this is not the optimal solution).
Would you recommend any other solution? - Is there a callback to the trigger of the warning "No pending request with requestId=..."
Thank you!

[kevinherron]

You don't really need to do much of anything. One of two things will happen:

1. you will eventually receive another PublishResponse, and if the sequence numbers aren't what is expected the client will start calling the Republish service to get any missed notifications.
2. if the Subscription/Server is broken, the client's subscription watchdog timer will eventually fire (onSubscriptionWatchdogTimerElapsed()) and let you know something is wrong with that Subscription.

[innaSigal]

@kevinherron thanks once again for your quick reply!

This scenario is very hard to reproduce and it happens rarely and also not on our setups. So I need a little bit more info:
In this scenario there is no recovery from this state and the application is restarted manually after 2 hours without data and/or keep alive being arrive.
I do see in Wire Shark that till the end of these 2 hours significant delays in the opcua read-write loops.
So from your description above It seems the onSubscriptionWatchdogTimerElapsed() callback is triggered.
Is there any special transaction in wire Shark I can identify for sure whether this callback is indeed finally triggered?
In Overriding onSubscriptionWatchdogTimerElapsed() is there any recommendation to do something different from the logic which is ran in onSubscriptionTransferFailed() to recover more quickly?
Thanks!

[kevinherron]

Unfortunately onSubscriptionWatcherTimerElapsed is not triggered by any specific event in Wireshark, but rather the absence of PublishResponses containing a NotificationMessage for a given Subscription.

I use this callback to log a warning and re-create the Subscription. Even just adding some logging would be a good place for you to start and see if this is your issue.

[innaSigal]

Hello @kevinherron,

I was able to "kind of" reproduce the bug using OPCUA server simulator.

I added a delay of 30 seconds to all the PublishResponses+keep alive on the OPCUA server side.
And I do see very similar behavior of OPCUA milo client - "No pending request" messages,
No keep Alive messages.

The difference is that using simulator I do see that After a 30 minutes there is a Warning in the log:
WARN org.eclipse.milo.opcua.sdk.client.subscriptions.OpcUaSubscriptionManager [milo-shared-thread-pool-110] [id=750008] expected sequence=500, received sequence=605. Calling Republish service...

But such warning doesn't appear in the logs of the real customer setups -
as described above, in those setups the failure occurs for ~2 hours after which the SW restarted manually.
During those 2 hours there is no such Warning of "Republish service" from milo.
*The milo version was never changed in our SW, so it same in both setups.

What can cause milo not to run the republish?

Thanks!

[innaSigal]

@kevinherron,

I also see that the onSubscriptionTransferFailed was implemented in 2021.
We are using eclipse milo sdk-client version 0.5.4.
How we are implementing the solution there?
We would like to avoid milo client version upgrade for now.
Is this possible?
Thanks,

[kevinherron]

Republish is only called once an out of sequence PublishResponse is received. If you never receive one, or the sequence was never incremented because data wasn't changing and only "keep alive" responses were being sent, then it won't get logged.

Further troubleshooting as to what and why would require logs and a long running Wireshark capture taken before/during/after the server not responding.

You'll need to upgrade to 0.6.x to get the subscription watchdog timer callback functionality. You should upgrade anyway, because there have been many bug fixes and workarounds implemented for broken servers since then. Old versions are not supported.

[innaSigal]

@kevinherron Thanks once again for the immediate support!

I tried to upgrade milo to the latest version from 0.5.4 and there is no backward compatibility to 0.5.4.
As this bug persists quite long tome in our SW and the customer wants a quick fix (we are in production mode in this context)
we are trying to avoid for now such a big change in our SW like upgrading the milo library and instead deliver a quick fix.
And we will upgrade milo for our next release.

The question is whether, in your opinion, there is some big risk in the next workaround: initiate OPCUA reconnection (disconnect from the OPCUA server and connect to it once again) if the last keep alive was received before a given amount of time.
We can also check when the automatic reconnection mechanism of milo client is triggered in case of OPCUA server total disconnection and will trigger the reconnection after this timeslot - so we will allow the milo client do the work in case of real server disconnection).

What are the trade offs using this solution and not upgrade the milo? Besides this bug introduced in some of the setups of our clients the OPCUA client is quite stable and we don't see any issues.

[kevinherron]

The question is whether, in your opinion, there is some big risk in the next workaround: initiate OPCUA reconnection (disconnect from the OPCUA server and connect to it once again) if the last keep alive was received before a given amount of time.

Yes this is okay, but just realize the onKeepAlive callback only fires when a keep alive notification is received, and that you should only expect keep alive notifications when none of the monitored items in a given Subscription is changing.

I guess the worst case is that you reconnect unnecessarily, but it's something to watch out for.

If you have some long running Wireshark captures that start before there are any issues (or, ideally start before you even make a connection) and then run until the server stops responding I'd be interested in looking at them. I might be able to tell you for sure if there's a bug in the client or server. Unfortunately it's often the server, because the burden of maintaining a Subscription and publishing keep alive and data change notifications is on the server. The client is really just listening for changes. That some servers end up having problems keeping their subscriptions running is why the subscription watchdog timer was introduced in 0.6.x.

[innaSigal]

@kevinherron hello once again,

We have a question regarding the OPCUA server bug and its solution using milo discussed previously:

1. The scenario is: the OPCUA server delays Publish Responses causing them arrive after the expected time out on the client(milo) side.
2. This includes responses to requests of type Keep Alive.
3. Milo detects this issue and starts health polling using requests of type Read Request.
4. Unfortunately the OPCUA responds to requests of type Read Requests immediately.
5. Based on 4 milo is not initiating automatic reconnection.
6. But milo triggers a callback which detects this scenario giving the opportunity for the user to reestablish connection.
7. As previous solution is not available in the old milo library we have we delivered currently a hot fix to the customer in which based on Add a Gitter chat badge to README.md #2 we are initiating reconnection in case of "frozen" keep alive for more than 2 minutes.
8. The problem with this solution is that in case the delay induced by the server on Publish Responses is not persistent, one keep alive message on time from the server is sufficient to update the last keep alive time to a recent time and "mask" the issue.
9. Based on this we would like to know whether the callback onSubscriptionWatcherTimerElapsed will resolve this type of issue once we are integrating the new milo library?
   Is it triggered based on "are there any time outs on Publish Requests" or "is there some Publish Response which is healthy" and then it is not triggered?

Thanks!