CONUS NRT V3 Schedule Jobs are Failing

[ranchodeluxe]

Problem

CONUS NRT V3 scheduled jobs are failing.

GH Actions showing which job ids are failing and here's are the docs to walk through to debug

Action Items

• figure out what's wrong and try to come up with a fix

[mccabete]

Love the write up!

seems like they are failing because pre-processed timesteps are missing. ie job 412d9f87-0ec3-438a-9315-fe5e45276a33

"FileNotFoundError('maap-ops-workspace/shared/gsfc_landslides/FEDSpreprocessed/NOAA20/20240323_AM.txt')

it looks like we do have a gap in our daily files at doy 83 - skipping from J1_VIIRS_C2_Global_VJ114IMGTDL_NRT_2024079.txt to J1_VIIRS_C2_Global_VJ114IMGTDL_NRT_2024085.txt

[mccabete]

As far as I can tell, the "daily" files are all released through the place we are querying from -- I can only see them going back a certain about of time, to June 3rd. https://nrt4.modaps.eosdis.nasa.gov/api/v2/content/archives/FIRMS/noaa-20-viirs-c2/Global/

Seems like they only distribute the last 60 days worth.

[mccabete]

The firms API seems to only be the last 10 days.

[mccabete]

Looks like it wasn't a data outage: https://ladsweb.modaps.eosdis.nasa.gov/alerts-and-issues/

[mccabete]

For a fix: I don't know. It seems like either we need to structure the code to be ok with "skipping" a few days OR we need to figure out how to derive our own .txt files from the data that we can get more reliably. Or both.

[ranchodeluxe]

For a fix: I don't know. It seems like either we need to structure the code to be ok with "skipping" a few days OR we need to figure out how to derive our own .txt files from the data that we can get more reliably. Or both.

Maybe it's acceptable for now (and 2024) to start our NRT runs on 2024-03-25 instead of 2024-01-01?

Then we'll be forced to deal with this during archival runs but that would be another data source

[mccabete]

Yeah as a temporary fix it's not the worst. We may run into other data holes, but I guess we can cross that bridge if we come to it.

[mccabete]

If this happens later in the season though (may onward) it will probably be chopping up real fires though.

[mccabete]

@ranchodeluxe is there any chance that V3 restarts every time there is a data gap?