iroute sometimes gets lost using dco-module

[mr-mister123]

Describe the bug
I have a server and a client previously running debian buster. The client is running in daemon mode and is always connected. i had a stable connection for years.
The clients network (192.168.89.0/24) is routed to the server by
a) giving it a fixed ip-adress within the vpn-subnet
b) and setting an iroute
in the client-file within the ccd-dir.

additionally the route-property (192.168.89.0/24) is set in the server.conf.

After upgrading both of them (didn't check what happens if only the server was upgraded) to bookworm using openvpn 2.6.3 and enabling the dco-module, behaviour changed as excepted:
since the dco-module is using kernel-routes (instead of the old internal routing-mechanism) i now have one route on the serverside, that is always set, if the server-daemon is running (resulting from the route-property):

192.168.89.0/24 via 192.168.23.2 dev tun0 metric 200

and one route, that is automatically beeing added if the client is connected:

192.168.89.0/24 via 192.168.23.253 dev tun0 metric 100

So far so god...

But: There is a strange behavior that seems to occur, if the client changes its public ip-address, for example through dhcp by the provider (i think, it has to do with that, but i'm not sure).

The route added by the dco-module (because of the iroute-command in client ccd) is getting lost.

ip route show

then doesn't list it anymore. The 192.168.89.0/24-network is then not reachable anymore.
But the vpn-channel seems to be alive since i can access the client through its static vpn-subnet-ip (192.168.23.253).

To Reproduce

• Setup a server using dco-module
• configure a client to route its subnet using iroute in ccd
• let a client connect and keep the connection alive until it's ip-address changes
• check kernel-routing-table on server-side

Expected behavior
The route created by the dco-module should not get lost

Version information (please complete the following information):
Server:

• OS: debian bookworm
• OpenVPN version: 2.6.3
• DCO: 0.0+git20230324-1

Client:

• OS: debian bookworm
• OpenVPN version: 2.6.3
• DCO: 0.0+git20230324-1

Additional context
I have an other client still running buster and therefore without dco enabled. this doesn't cause any problems. but it doesn't change its ip-adress. I dont know if it is related to that or if it makes a difference, when the client is using the dco-module. this just as a hint to you...

thanks for your effort and greetz,
Karsten

[ordex]

Hi @mr-mister123 and thanks a lot for your report.
Any chance you could provide a server log with --verb 6 so we can check what the server is exactly doing and why/when that route is eliminated?

As a side note, do you know where 192.168.23.2 is coming from in the other route?

[mr-mister123]

Hi,

As a side note, do you know where 192.168.23.2 is coming from in the other route?

This route is beeing added, if 'route' is set in the server.conf.
Man-page states:

However, when using DCO, the --iroute directive is usually
enough for DCO to fully configure the routing table. The extra
--route directive is required only if the expected behaviour is
to route the traffic for a specific network to the VPN interface
also when the responsible client is not connected (traffic will
then be dropped).

That's exactly what i want to achive (drop packets to 192.168.89.0/24 if client is not connected). Already tested without that route-directive, but the route added by dco-module still get's lost sometimes.

Any chance you could provide a server log with --verb 6 so we can check what the server is exactly doing and why/when that route is eliminated?

I've activated verb 6, now i have to wait for the route to get lost. this can take a while. i hope, i'll manage it at all, since i do not monitor that route the whole time... and (somehow crazy) it get's repaired 'by self': it was broken yesterday, but for example now it's fixed although i didn't change anything (neither server nor client-daemon were restarted).
Maybe it 'toggles'?!

• fresh connection: everything is okay, route is there
• client changes ip-adress: route get's lost
• client changes ip-adress the next time: route get's added again (maybe, because it was'nt there before)

i think, this is a race-condition.
cause of the ip-adress-change the server closes connection. simultaniously the client starts a new connection. on connect the route is still there and no new route will be added. but the close of the old connection is still pending. after that the rout gets removed an now it's missing.. can this be?

[ordex]

it could be related to overlapping events. yes. This is exactly what I am hoping to see in the log, because it' the openvpn daemon that installs/removes routes and the daemon is a running a single thread. So the log should be fairly clear about what is going on.

[cron2]

Hi,

On Wed, Oct 25, 2023 at 01:12:16AM -0700, Antonio Quartulli wrote: it could be related to overlapping events. yes. This is exactly what I am hoping to see in the log, because it' the openvpn daemon that installs/removes routes and the daemon is a running a single thread. So te log should be fairly clear about what is going on.

Yeah, something like - client connects, /24 iroute is installed - client reconnects from a new IP, but OpenVPN server does not know "it is gone!" yet, so the new connection kills the old TLS instance - new iroute is installed, "route already exists" - cleanup of old instance kills the (one and only) iroute this is never a problem without DCO (as iroutes are not formally removed, they just disappear) but with the explicit route removal call things might break. gert -- Gert Doering - Munich, Germany ***@***.***

[mr-mister123]

Hi,

finally it happend ;-)

now i have the state, that the route is missing, but connection seems to be active. i can reach the client through its fixed vpn-ip-address 192.168.23.253, but not through the routed 192.168.89.0/24-net...

here is the log:

openvpn.log

the route got lost at 2023-10-28T06:16:08, re-connected at 2023-10-28T07:09:22. but no new route was created...
interestingly it's quite a long time-range between route-deletion and reconnection, so the problem doesn't seem to be related to overlapping events... ?

Greetz,
Karsten

[ordex]

Thanks. This is indeed interesting.
And this is not a disconnection and reconnection, we rather have a rekeying timeout which ends up with OpenVPN "restarting" the client instance (this is when the route is removed). Later (at 07:09:22) the client performes a new renegotiation which works as expected. However, this is not a full reconnection, therefore routes are not added again.

I am leaning towards "upon instance restarting we should not delete the route, because the instance is still there"

[ordex]

Although I don't understand why the client is switching from peer-id 2 to peer-id 1, despite not going through a real reconnection..