Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Process JEDI radiance files as a single tarball #2862

Open
RussTreadon-NOAA opened this issue Aug 27, 2024 · 3 comments · May be fixed by #2875
Open

Process JEDI radiance files as a single tarball #2862

RussTreadon-NOAA opened this issue Aug 27, 2024 · 3 comments · May be fixed by #2875
Assignees
@RussTreadon-NOAA
Labels
feature New feature or request

Comments

@RussTreadon-NOAA
Copy link
Contributor

What new functionality do you need?

At present JEDI radiance bias correction uses three files per satellite_sensor. This result in many individual files to move, copy, link, archive. While JEDI applications need to work with individual files, file operations are more easily applied to a single file.

This can be achived by tar'ing all JEDI radiance bias correction files into a single file. This single file is what gets moved, copied, linked, and archived. Individual radiance bias correction files can be obtained by untar'ing the radiance tarball in the run directory of specific jobs.

What are the requirements for the new functionality?

Update JEDI DA scripts which currently process individual radiance bias correction files to process a single radiance tarball.

Acceptance Criteria

Once work for this issue is complete, the COM directory will contain a single radiance bias correction file per cycle. Analysis results when using this tarball should be identical to results obtained by using individual radiance bias correction files.

Suggest a solution (optional)

No response
@RussTreadon-NOAA RussTreadon-NOAA added feature New feature or request triage Issues that are triage labels Aug 27, 2024
@RussTreadon-NOAA RussTreadon-NOAA self-assigned this Aug 27, 2024
@RussTreadon-NOAA RussTreadon-NOAA mentioned this issue Aug 27, 2024
Merged
7 tasks
@RussTreadon-NOAA
Copy link
Contributor Author

Work for this issue will be done in RussTreadon-NOAA:feature/radbcor

@RussTreadon-NOAA RussTreadon-NOAA mentioned this issue Aug 27, 2024
Closed
@CoryMartin-NOAA CoryMartin-NOAA mentioned this issue Aug 28, 2024
Open
7 tasks
@RussTreadon-NOAA RussTreadon-NOAA mentioned this issue Aug 28, 2024
Closed
4 tasks
RussTreadon-NOAA added a commit to RussTreadon-NOAA/global-workflow that referenced this issue Aug 28, 2024
@RussTreadon-NOAA
replace separate JEDI radiance bias correction files with tarball (NO…
20fb88c 
…AA-EMC#2862)
@RussTreadon-NOAA
Copy link
Contributor Author

Hera g-w C96C48_ufs_hybatmDA CI

Install feature/radbcor at 20fb88c on Hera. Set up g-w CI for C96C48_ufs_hybatmDA.

Note, for these test specify ICSDIR_ROOT=/scratch1/NCEPDEV/global/Russ.Treadon/data/ICSDIR. This is copy of the glopara ICSDIR_ROOT with the satbias and tlapse radiance bias correction files for C96C48_ufs_hybatmDA replaced with a single gdas.t18z.radbcor tarball.

All enkfgdas, gdas, and gfs DA jobs successfully run to completion. Only 20240224 00Z gfs forecast and post jobs remain running.

The enkfgdas, gdas, and gfs init jobs extracted satellite_sensor radiance bias correction files from the radbcor tarball. The final job created a radbcor tarball with updated coefficient and copied the file to the appropriate COM directory. The gdas archive job included the radbcor tarball in gdas_restarta.tar.

This test will be repeated using the official ICSDIR_ROOT once issue #2869 is closed.

@RussTreadon-NOAA
Copy link
Contributor Author

GDASApp PR #1267 updates the drivers for the atmospheric local ensemble and variational ctests to process the radiance bias correction tarball instead of individual satellite_sensor bias correction files. The sorc/gdas.cd hash in g-w feature/radbcor will be updated a new GDASApp hash once PR #1267 is approved and merged into GDASApp develop.

@RussTreadon-NOAA RussTreadon-NOAA linked a pull request Aug 28, 2024 that will close this issue
Open
10 tasks
RussTreadon-NOAA added a commit to RussTreadon-NOAA/global-workflow that referenced this issue Aug 29, 2024
@RussTreadon-NOAA
rename JEDI radiance bias correction tarball file to be more self-des…
73aa233 
…cribing (NOAA-EMC#2862)
@WalterKolczynski-NOAA WalterKolczynski-NOAA removed the triage Issues that are triage label Sep 3, 2024
RussTreadon-NOAA added a commit to RussTreadon-NOAA/global-workflow that referenced this issue Sep 7, 2024
@RussTreadon-NOAA
update gdas.cd hash to include recent commits to GDASApp develop (NOA…
1203866 
…A-EMC#2862)
RussTreadon-NOAA added a commit to RussTreadon-NOAA/global-workflow that referenced this issue Sep 9, 2024
@RussTreadon-NOAA
update pygfs jedi class and atmospheric tasks to extract JEDI radianc…
5985e9f 
…e bias correction files from tarball (NOAA-EMC#2862)
RussTreadon-NOAA added a commit to RussTreadon-NOAA/global-workflow that referenced this issue Sep 10, 2024
@RussTreadon-NOAA
initial attempt to generalize processing of variational bias correcti…
666d7d6 
…on files using jedi class (NOAA-EMC#2862)
RussTreadon-NOAA added a commit to RussTreadon-NOAA/global-workflow that referenced this issue Sep 10, 2024
@RussTreadon-NOAA
move VarBcDir inside initialize_analysis method (NOAA-EMC#2862)
1e4a284
RussTreadon-NOAA added a commit to RussTreadon-NOAA/global-workflow that referenced this issue Sep 11, 2024
@RussTreadon-NOAA
update sorc/gdas.cd hash to bring in radiance tarball ctest changes (N…
b48e126 
…OAA-EMC#2862)
RussTreadon-NOAA added a commit to RussTreadon-NOAA/global-workflow that referenced this issue Sep 12, 2024
@RussTreadon-NOAA
add extract_tar method to jedi class, use extract_tar in atm and atme…
ba77d4e 
…ns analysis scripts (NOAA-EMC#2862)
RussTreadon-NOAA added a commit to RussTreadon-NOAA/global-workflow that referenced this issue Sep 17, 2024
@RussTreadon-NOAA
improve error handling for jedi class tarfile.extractall (NOAA-EMC#2862)
0ab7bdd
RussTreadon-NOAA added a commit to RussTreadon-NOAA/global-workflow that referenced this issue Sep 17, 2024
@RussTreadon-NOAA
remove unnecessary close following tarfile add (NOAA-EMC#2862)
8b627a1
RussTreadon-NOAA added a commit to RussTreadon-NOAA/global-workflow that referenced this issue Sep 17, 2024
@RussTreadon-NOAA
use endswith to clean up scripting (NOAA-EMC#2862)
eb4fc37
RussTreadon-NOAA added a commit to RussTreadon-NOAA/global-workflow that referenced this issue Sep 18, 2024
@RussTreadon-NOAA
refactor jedi class method extract_tar (NOAA-EMC#2862)
ed52b72
RussTreadon-NOAA added a commit to RussTreadon-NOAA/global-workflow that referenced this issue Sep 18, 2024
@RussTreadon-NOAA
update sorc/gdas.cd to current head of GDASApp develop (NOAA-EMC#2862)
80305b6
RussTreadon-NOAA added a commit to RussTreadon-NOAA/global-workflow that referenced this issue Sep 18, 2024
@RussTreadon-NOAA
make jedi method extract_tar static, clean up comments, correct typo (N…
821ee76 
…OAA-EMC#2862)
RussTreadon-NOAA added a commit to RussTreadon-NOAA/global-workflow that referenced this issue Sep 19, 2024
@RussTreadon-NOAA
correctly prefix tlapse radiance bias correction files for cycling (N…
042ec50 
…OAA-EMC#2862)
RussTreadon-NOAA added a commit to RussTreadon-NOAA/global-workflow that referenced this issue Sep 20, 2024
@RussTreadon-NOAA
extend duration of C96C48_ufs_hybatmDA CI by one cycle to 2024022406 (N…
5306b06 
…OAA-EMC#2862)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@RussTreadon-NOAA RussTreadon-NOAA

Labels
feature New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Replace separate JEDI radiance bias correction files with tarball RussTreadon-NOAA/global-workflow
2 participants
@RussTreadon-NOAA @WalterKolczynski-NOAA