[Processing Request]: CA DISP-S1 Frames

[gracebato]

Venue

PST

Product

DISP-S1

SAS Version

No response

SDS Version

No response

Input Data

asc_frameIDs_CA.json
des_frameIDs_CA.json

Share Results

• Google Earth Engine (GEE)
• NASA Distributed Active Archive Centers (DAAC) User Acceptance Testing (UAT)

Additional Notes

TBC by David B. if request goes to UAT.

[LucaCinquini]

Yes to pushing data to ASF UAT.
Parameters:
2016-07-01 to 2018-07-01
k=15, m=5

[LucaCinquini]

[philipjyoon]

Processing has started. Running on INT-POP1 cluster; these will be delivered to ASF UAT.

[philipjyoon]

Using these two batch files in PCM
asc_california.json
des_california.json

[philipjyoon]

f3328 needs to be reprocessed using the DISP-S1 burst database s3://opera-ancillaries/disp_frames/disp_s1_consistent_burst_db/opera-disp-s1-consistent-burst-ids-2016-07-01_to_2024-09-04.json

This new burst file contains less bursts for that frame to use many more sensing times.

There are likely a few other frames that fall into this and they will all need to be reprocessed after the current processing is completed. This is a one-time event due to transitional period of this historical database file. We shouldn't have these issues in future.

[philipjyoon]

Processing is mostly complete and here is the result. This is how many sensing dates that were processed as part of historical processing per frame. For example frame 46288 was processed for 30 sensing dates (two runs with k=15) and so on.

As mentioned above f3328 needs to be processed again. We can see that it didn't process at all here. While f24728 also shows 0 here it is actually correct because there weren't 15 sensing dates for that frame in the first two years.

{'46288': 30, '46289': 15, '26690': 30, '26691': 15, '38500': 30, '38501': 15, '38502': 15, '38503': 45, '38504': 45, '38499': 15, '26689': 15, '3328': 0, '46290': 15, '3327': 45, '3326': 30, '3325': 15, '18902': 45, '30714': 45, '18903': 15, '30713': 15, '11113': 15, '18904': 30, '30716': 45, '11112': 45, '18905': 45, '30715': 45, '11114': 30, '11117': 15, '30712': 45, '46291': 30, '18901': 45, '30711': 45}

{'44326': 45, '44325': 45, '36540': 45, '36541': 45, '9160': 30, '28757': 30, '44328': 30, '44327': 45, '28758': 15, '28759': 15, '44329': 15, '16939': 60, '36544': 15, '36545': 30, '36546': 30, '9157': 30, '24726': 15, '9158': 15, '9155': 45, '24728': 0, '9156': 45, '24727': 15, '9159': 30, '9154': 30, '36539': 45, '28760': 30, '16943': 45, '16942': 45, '16941': 45, '16940': 60, '16946': 45, '16945': 30, '16944': 45}

[philipjyoon]

The PCM team still needs to implement the “end of frame series behavior” into the historical processing code. This is the behavior on the last n number of sensing dates when it’s less than k=15. Historical processing executes every k sensing date so there is often remainder sensing dates at the end. The behavior will be to either a) run one single historical processing if n >= k/2 or b) run n number of reprocessing jobs if n < k/2. In order to process up to the very end of the date range, this feature needs to be implemented and then the historical jobs needs to continue to completion.

Asked PST whether they'd like this also executed or what we have is sufficient.

[philipjyoon]

Correction: 25 frames are still processing now including f3328

[philipjyoon]

Processing is complete.

Ascending:
{'44326': 45, '44325': 45, '36540': 45, '36541': 45, '9160': 30, '28757': 30, '44328': 45, '44327': 45, '28758': 30, '28759': 30, '44329': 45, '16939': 60, '36544': 30, '36545': 30, '36546': 30, '9157': 30, '24726': 15, '9158': 30, '9155': 45, '24728': 0, '9156': 45, '24727': 15, '9159': 30, '9154': 30, '36539': 45, '28760': 30, '16943': 45, '16942': 45, '16941': 45, '16940': 60, '16946': 45, '16945': 45, '16944': 45}

Descending
{'46288': 30, '46289': 30, '26690': 45, '26691': 45, '38500': 45, '38501': 45, '38502': 45, '38503': 45, '38504': 45, '38499': 45, '26689': 45, '3328': 30, '46290': 30, '3327': 45, '3326': 45, '3325': 45, '18902': 45, '30714': 45, '18903': 45, '30713': 45, '11113': 45, '18904': 45, '30716': 45, '11112': 45, '18905': 45, '30715': 45, '11114': 45, '11117': 45, '30712': 45, '46291': 30, '18901': 45, '30711': 45}

[philipjyoon]

Frames 9158 ran into an intermittent error. It's being re-tried to continue processing right now.

[asjohnston-asf]

Ascending frames 36542/36543 and descending frames 11115/11116 were included in the original asc_frameIDs_CA.json / des_frameIDs_CA.json files, but not in the asc_california.json / des_california.json batch files. Was that intentional?

[philipjyoon]

@asjohnston-asf 11115 and 11116 were processed as part of this other request: #53 It was performed from a different OPERA venue and those were not sent to ASF. The PST team still has access to that data.

36542 is being processed as part of this ticket currently: #55

I'm not sure what the story with 36543 is. I believe there was a good reason why we didn't process but do not recall it. IF this is an issue, please contact @gracebato and she can direct me on how to proceed.

[asjohnston-asf]

Thanks! Per @forrestfwilliams , missing those frames shouldn't impact ASF as we move forward with development, so no need for any extra action on our behalf.

[asjohnston-asf]

Per @forrestfwilliams, further development by ASF on the displacement/velocity map layers depends on this data set being reprocessed to the most recent version (v0.7?) and ingested at ASF, specifically to enable:

• masking via the new "recommended_mask" layer rather than the connected components layer
• adding the ability to generate map layers from multiple input granules that include a change in reference date

While we'd like to validate our work with the entire ascending and descending datasets, we could start our work with as few as four contiguous frames from a single orbit direction, say:

• 36541
• 36540
• 16942
• 16941

If you could sequence work to deliver those four frames first, that would help us start work earlier.

[arothjpl]

Processing has started on 10-30-24. Running on INT-POP1 cluster; these will be delivered to ASF UAT.

[philipjyoon]

Request 54 processing has started using the following batch_procs using OPERA PCM 3.1.0 FINAL

Request_54_asc_california.json
Request_54_des_california.json

[philipjyoon]

Processing has been stopped due to a SAS bug

[LucaCinquini]

What SAS bug?

[philipjyoon]

@LucaCinquini Scott S stated the following on PST channel this afternoon:

I spot checked this- I’m sorry that these the v0.8 SAS fix did not fix the short wavelength bug and recommended mask being all zeros.
I’ve found the (dumb) bug in the SAS code causing this, so i’ll be merging a fix shortly.