Potential issues with RS41 decoding with no GPS lock

[darksidelemm]

Describe the bug
There's been a lot of GPS jamming observed on radiosondes out of Finland lately (well, really all over the place, but I'll use the Finland launches as an example).
If you take a look at the plots for one of these sondes, you see some interesting stuff:
https://grafana.v2.sondehub.org/d/bbaa7894-e5f4-4c0d-be96-897b4ffde43b/radiosonde-telemetry-dashboard?var-Serial=V3450224&from=1698390411000&to=1698397668131&orgId=1

As the sonde ascends, the altitude appears to become 'fixed', as does the number of sats reported. At the time this happens, all the auto_rx stations stop reporting. Investigation of the logs from stations in the area shows that the sonde is actually reporting 0 sats, and so auto_rx is dropping the telemetry frames.

rdz_ttgo_sondes seem to continue to upload data, but with frozen positions, altitudes, and SV info. The PTU data is still valid however!

While I haven't delved too far into the code, I did notice this line: https://github.com/dl9rdz/rdz_ttgo_sonde/blob/devel/RX_FSK/src/RS41.cpp#L456
which looks like if the number of sats is < 4, it doesn't update your RS41 telemetry store, so i expect the previous >4 value will be in there still.

To Reproduce
Keep an eye on launches out of Finland on sondehub.org!

Expected behavior
If the RS41 is reporting 0 sats, then this should be reflected in what is uploaded to SondeHub, not some other previous value. SondeHub will filter out positions with 0 SVs and not show them on the map, but they will still end up in the database, so PTU data can be shown on the Grafana plots.

Screenshots
Refer plots example: https://github.com/dl9rdz/rdz_ttgo_sonde/blob/devel/RX_FSK/src/RS41.cpp#L456

Version and specific configuration:
Data observed from v0.9.3, and devel 20230829

Additional context
GPS jamming makes me sad :-(

[j0uni]

I've been closely monitoring and decoding these sondes here in Finland.

What is happening is that when the GPS interference is occuring, the Sonde stops sending frame 7B completely (the ECEF information). It continues sending most of the other frames during this GPS "blackout".

Easy solution for rdzsonde would be make this 7B frame "timeout" after certain period and reset the location and altitude and SV to zero. It is really good that rdz continues sending decoded data out because there is no reason it should stop sending even the location of sonde is missing.

I think auto_rx needs fix too; it should send the packets even the GPS location is lost. There are other valuable information that is nice to see on the Grafana and on the map interface (temperature, pressure). On the other hand because auto_rx stations stop reporting in this situation, it has been really easy to spot the moment from grafana :)

Also my suggestion is that sondehub should accept (as it has) these non-location packets and continues showing rest of information with map. But I would change the sonde icon to some ? question mark or something to show that the location is uncertain and show that ? icon at the last known place (the last frame number).

Edit: during the inteference sondes start to jump forth and back on sondehub map. This is because rdzsonde stations are sending "old" data. And as the stations might miss the newest packets, the jumping seems to be quite strange. This could be fixed so that sondehub only shows position of the frame that has the highest frame count.

[j0uni]

@darksidelemm BTW, what do you mean by "(well, really all over the place, .." - do you mean all world sondes are being affected by this or many regions? Can you share where else the same effect is occuring?

[ghgeiger]

I'm not sure if it's related, but I witnessed the sonde launched last evening (10/26 23:00UTC) from Nashville Tennessee, US, stop all reporting just above 28,000 m. I thought it was interesting at the time, but it's concerning if it has the same root cause as the ones in the above discussion.

This was sonde DFM17 23027668

[j0uni]

I'm not sure if it's related, but I witnessed the sonde launched last evening (10/26 23:00UTC) from Nashville Tennessee, US, stop all reporting just above 28,000 m. I thought it was interesting at the time, but it's concerning if it has the same root cause as the ones in the above discussion.

During GPS inteference sonde works otherwise totally normally, it just doesn't send GPS position related information. Most of the sondes get GPS fix again (after the inteference period) and transmit position for rest of the flight. Some might loose their fix and even the inteference ends the sonde might still not get fix (probably because fast vertical speed during descent).

I would guess your case is just sonde malfunctionality.

[ghgeiger]

I just compared with the data from the sonde in your first post & you're right. Looking at the reports from autorx stations, it appears that this sonde either had a malfunction transmitter or its battery was dying early.

[j0uni]

Here is one DFM09 that had GPS inteference btw. I don't have direct data from the sonde but based on Grafana the sonde seems to stop transmitting during interference: https://grafana.v2.sondehub.org/d/bbaa7894-e5f4-4c0d-be96-897b4ffde43b/radiosonde-telemetry-dashboard?var-Serial=635156&from=1698235996000&to=1698244613000&orgId=1

In Finland we have only Vaisala RS41-SG(P) sondes so my previous comments were of Vaisala sondes. Those send data during interference.

[darksidelemm]

OK, a few things to unpack here:

• Areas of GPS interference: Common places at the moment include near Ukraine and Israel. Now also observed on sondes from Finland.
• The RS decoders auto_rx uses sends a report of sats=0 when the RS41 has no GPS lock. This might be because the internal data store gets reset on each new packet? Unsure. Either way the sats number ends up at 0 and then auto_rx drops the packet internally.
• I agree uploading data even with no GPS lock is potentially valuable, and I will likely move towards this in auto_rx, however some care needs to be taken here not to upload packets where the reported date/time is clearly way off. It's likely the sonde will continue to maintain time once it has lock.
• The SondeHub Tracker is location based. If there's no valid position we're not going to do anything with it on the Tracker. However, you can always go to the Grafana plots to see plots of other data (like the plot I linked).

Anyway, based on the information that the 7B frame isn't being transmitted, then I would suggest that the appropriate behaviour is:

• Check the reported time from the sonde (This is still sent I believe?). If this time is more than a few minutes out from your system time (dunno how you'll handle this on the RDZ...) then discard the packet, as this might be data from a sonde that has only just been turned on and has never gotten GPS lock.
• If the time is OK, clamp the lat/lon to 0.0/0.0 (or leave it as the last known location, though I think 0.0,0.0 is less misleading - only provide the data that you have from the packet!), and set sats=0, then upload the frame.

The SondeHub Tracker will drop the position, but the data will still be available for plotting in Grafana.

[darksidelemm]

I've raised an issue on the auto_rx repo ( projecthorus/radiosonde_auto_rx#831 ) about handling this case. @j0uni some kind of recording of a sonde that has lost gps lock in flight would be useful to confirm the behaviour of the auto_rx decoder in this situation.

[j0uni]

@darksidelemm some comments to you:

"The SondeHub Tracker is location based. If there's no valid position we're not going to do anything with it on the Tracker."

I agree. But it would be super cool to see these sondes that have lost GPS lock on map on one glance. Question mark or "sad sonde" or something visual would be perfect.

"If the time is OK, clamp the lat/lon to 0.0/0.0 (or leave it as the last known location, though I think 0.0,0.0 is less misleading - only provide the data that you have from the packet!), and set sats=0, then upload the frame."

Maybe I didn't get the idea completely but rdzsonde continues sending old lat/lon when GPS location is lost. This of course could be fixed with for example the "timeout" idea I wrote previously. I think rdzsonde doesn't need anything else to be fixed in this case - just indicate that sonde has lost its location.

However even if rdzsonde is fixed today, there will be stations that will continue using old firmware for long time and the "problem" persist - meaning "sonde jumping around on sondehub". This is purely caused by different rdzsonde stations each having different "old" location. This could be fixed in sondehub by just showing the position of highest frame number. Then the icon would not move back in time.

I'm currently recording raw data from V1631304 and will send you the file when the flight has been finished. Well .. assuming it loses GPS fix which I'm pretty sure it will.

[darksidelemm]

Showing sondes that have lost GPS lock is possible, but difficult. Something for a later date, i've got enough changes to make at the moment!

My philosophy with data uploaded to SondeHub is that stations should only be sending data that is 'current'. e.g. if they aren't getting a position from the current data frame from the sonde, then they should not be sending an old position in place of that missing data. Since you have to send some kind of position for SondeHub to accept the data, send 0.0/0.0.

We can also get around the old rdz_ttgo_sonde version issue by just blocking traffic from old versions once a fix is available. We've already recently done this with DFM telemetry. It would be nice if the errors we send back were made visible to rdz_ttgo_sonde users so they know why we are blocking their data, but that's a separate issue.

[j0uni]

"My philosophy with data uploaded to SondeHub is that stations should only be sending data that is 'current'. e.g. if they aren't getting a position from the current data frame from the sonde, then they should not be sending an old position in place of that missing data. Since you have to send some kind of position for SondeHub to accept the data, send 0.0/0.0."

I think 0.0/0.0 is a great plan. Easy to identify and easy to "block".

Blocking older rdz versions would be bad idea I think. So many stations would be left out around the world. I think this should be fixed with option that causes less damage.. But I think it could be good idea to implement some kind of "alarm" text on rdz web UI that can show information from sondehub, such as notifying of too old version.

Here is raw data from V1631304 recorded wih Auto-RX. This sonde had very typical inteference pattern so this is a very good example what is happening.

20231028-053604_RS41_403000000.raw.zip

[dl9rdz]

I have some status information (mainly on eph fetch status for the old RS92) already on the "About" page of the web interface, and I intend to add more information there (status of all export connection, I can also show the last response message from sondehup api there). I plan to work on that in two weeks from now, and will consider all suggestions I get on this by then.

[darksidelemm]

Thanks @dl9rdz. I don't think this is that urgent, it's reasonably obvious that something is odd, as the balloon stops moving. Unfortunately without the true indication of number of sats its difficult to write anything to alert the user that something is wrong however.

[darksidelemm]

Having dug into the raw data from @j0uni a little more, it looks like the position values (as ECEF) freeze when the sonde loses gps lock, and the numSV field ('sats' when uploaded to SondeHub) reports as 0, indicating loss of lock. Time continues to advance.

Given the sonde is reporting a position, I would consider continuing to report that position, though ensuring the 'sats' field is also reported correctly (0, so that the tracker can handle the data appropriately) to be a valid approach.

[j0uni]

@darksidelemm Do you mean by freeze that sonde keeps sending 7B with old values ? I'm pretty sure that it didn't do that during inteference. Need to recheck that.

EDIT: checked already. Yes it actually does that and works like you have described. The good part is that the sat count drops to zero.

[darksidelemm]

Yep - I think that's the key thing that needs to be passed on correctly, which rdz_ttgo_sonde does not appear to be doing at the moment.

[darksidelemm]

We're still seeing this issue on multiple sondes.

[hr-ru]

The decoder does not update sats info, that is easy to fix.

The current sondehub code does not report anything if sats<4 (which will actually do that after fixing the problem that this info was not updated), so nothing will be reported. Or do you prefer frames to be uploaded with sats=0, given that other information in the frame could be useful?

[darksidelemm]

Yep, as long as the sats field is reported as 0, the data won't show up on the tracker, but will be accessible via grafana and directly from the database.

…

On Sat, 10 Feb 2024, 09:17 Hans P. Reiser, ***@***.***> wrote: The decoder does not update sats info, that is easy to fix. The current sondehub code does not report anything if sats<4 (which will actually do that after fixing the problem that this info was not updated), so nothing will be reported. Or do you prefer frames to be uploaded with sats=0, given that other information in the frame could be useful? — Reply to this email directly, view it on GitHub <#390 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAH57E5SHR5MINV4DY6JDXDYS2RRXAVCNFSM6AAAAAA6SQDA42VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZWG4YDIMRSGY> . You are receiving this because you were mentioned.Message ID: ***@***.***>