Ephemeral Jetstream subscription (KeyValueWatch) does not seem to reconnect when nats restarts.

[GrafBlutwurst]

Hi everyone,

I am observing heartbeat alerts on the inbox subscription for my key value watch after nats restarts. After some breakpoints and some wireshark I think it's due to that fact that it's an ephemeral consumer and the consumer is not recreated on a reconnect. Rather only the subscription to the delivery inbox is resubmitted.

That also tracks with the what we see using the nats cli when looking at consumers for the KV store which after the restart and successful reconnect sits still at 0.

Is this intended behaviour and we're using the library wrong or is there an issue with ephemeral JS consumers?

[scottf]

So the behavior for the ephemeral is correct. If the server restarts, the ephemeral is gone. The behavior during kv should be that the watcher makes a new subscription. I'll take a look into this.

[GrafBlutwurst]

yep server side behaviour is as intended, it's the client consumer handling that seems off to me. Correct me if I'm wrong (I don't know the ins and outs of the nats/jetstream protocol yet) but here's what I think is happening:

On a reconnect we loop through all subscriptions (sync ones held on connection directly, and async ones held by dispatchers) and generate resubscribe messages:

1. for non jetstream subscriptions this just works because they need no jetstream consumer creation
2. for durable jetstream subscriptions it works because the consumer is ... well durable
3. for ephemeral jetstream subscriptions it fails because we only subscribe to the inbox subject but do not recreate the consumer. Presumably this generates heartbeat alerts because it's the consumer creation that causes the server to emit heartbeats for that consumer/subscriptions.

Edit: So without having fully verified this I think the issue might not even be specific to KV watcher but rather to ephemeral jetstream subscriptions because during the reconnect the possibility of a server restart (i.e. ephemeral consumers are gone) is not handled.

[scottf]

Except for ordered consumer, which specifically tracks which stream sequence was received last, ephemerals will not resume, because they don't know where to start. The best thing is to be aware of the the event listener. You might need to track the ephemeral state yourself, to start a new ephemeral where you left off. An ordered consumer (see subscribe options) is a good option if you don't need to ack the messages.
Part of the question is where to draw the line on doing the work for the developer in the client.

[GrafBlutwurst]

Yeah that's a good point, but in that case I think more options need to be available in the creation of a key value watch. The subscriptions options are hard coded and not parameterizable.

Edit: also the KV watch is an ordered consumer iirc. So it would be applicable?