datalink packet ids

[crotwell]

new issue (from #60)

Conern that v4 ringserver cannot be drop in replacement for prev version due to auto-upgrade of packet ids to fdsn sourceid style for datalink. While I like new style source ids and believe they should be used, the auto-migration of ids (I think) means that while data input to new ringserver is ok regardless of how the client creates ids, the datalink output side breaks backwards compatibility.

Should old datalink clients be able to connect to get data without changing their MATCH pattern?

For datalink with old style ids it doesn't find any streams. So this works:

Send: MATCH FDSN:CO_JSC_00_H_H_./MSEED 
start datalink at : 2024-12-15T14:20:12.776Z 
Send: POSITION AFTER 1734272412776000 undefined 
Send: STREAM

but this fails to match any streams:

Send: MATCH CO_JSC_00_HH./MSEED 

The release notes sound like old style match patterns for datalink should still work. Am I misunderstanding?

All legacy stream IDs for miniSEED of the form: NN_SSSSS_LL_CCC/MSEED will be converted to FDSN Source ID form: FDSN:NN_SSSSS_LL_C_C_C/MSEED.

[chad-earthscope]

You are not misunderstanding. Version 4 is a new major version that I've tried to make as drop-in-replacement as possible with DataLink stream IDs being an exception.

In the general, ringserver stream IDs are arbitrary with a convention for miniSEED records. Adopting and providing a transition for new, standard-based convention from the old convention makes a lot of sense. Since SeedLink does not directly select stream IDs, the SeedLink selection syntax can be transparently mapped to the new miniSEED convention, which is much easier when the ID convention is consistent (thus the translation of old to new on insertion). DataLink, OTOH, directly matches stream IDs and is not trivially adapted to the new convention. The use case of ringserver as miniSEED-for-SeedLink received the most consideration regarding backwards compatibility while moving to a new convention. I'm sure you agree this makes sense.

So why didn't DataLink get a transparent path to the new convention? This is mostly due to my estimation that the vast majority of use cases do not include using DataLink as a consuming client, combined with the complexity of building that new path. You are a noted exception to using DataLink as a consuming client and you've promoted that path with supporting software (which I appreciate!). I'm hopeful that most users and use cases of ringserver will not be effected in this major version change.

I'd like to provide a transition path for DataLink clients selecting streams using the old, ringserver-specific, convention if I could figure out how. I've considered these options:

1. translate the submitted MATCH/REJECT regexes to the new convention when they match the old (which is what is happening for SeedLink). This would be really difficult to get right. I'm doubt it's even possible as I expect there are many, many ambiguities in trying to convert arbitrarily complex regexes to a new form.

2. restructure the ring buffer to allow streams to be identified with multiple IDs, likely by changing fundamental stream IDs to UUIDs/hash values and some lookup to match 1-N text IDs. This is non-trivial complexity.

Fundamentally it comes down to "is this worth the work and complexity". If I could find an easy way, or a lot of spare time, I'd make this work. Other suggestions welcome.

P.S. Note on the option to restructure the ring buffer (2) for future consideration: replace the internal stream index (the streamidx file) with an sqlite3 DB. Including an extra table that translates stream UUID/hash to text stream IDs would be relatively straightforward. This is a non-trivial change.

[chad-earthscope]

There are other considerations for DataLink clients. If the selection was translated would the packet be returned as the translated ID or one of it's alternates?

Addendum: and all of this consideration goes towards drop-in replacement, and is counter-productive to moving towards the standard ID usage as fast as possible.

[crotwell]

Humm, yes rewritting a regex would be challenging. I'm inclined to agree with the breaking-compatibility decision.

Change of question, how can a client detect if IDs should be old style or new FDSN style? I would like to create example clients that will work today, but also will not break on the day rtserver.iris upgrades to v4. Is there something in the initial ID handshake that would trigger this for a client that wished to interoperate with both old and v4 ringserver,ie it knows station and channels and wishes to pass in correct MATCH regex for them? Is keying off of something in ID best, like:

DataLink v1.1 (RingServer/4.0.0) :: DLPROTO:1.1 PACKETSIZE:512 WRITE

vs

DataLink 2018.078 :: DLPROTO:1.0 PACKETSIZE:512

Guessing the DLPROTO:1.1 is best? Or would it be worth having an additional "capability" in the ID line to indicate ID style? Not sure there is an easy way for a client to create a regex that works in both cases. Regardless, documenting an upgrade path for clients would be helpful.

BTW, what is difference between DLPROTO:1.1 and DLPROTO:1.0? I only see docs for the 2008 version here:
https://earthscope.github.io/libdali/datalink-protocol.html
Would be nice if the protocol docs could be updated to show 1.0 vs 1.1 differences.