Add fire blending to GOCART #2883

bbakernoaa · 2024-08-30T16:01:32Z

Description

Type of change

Bug fix (fixes something broken)
New feature (adds functionality)
Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this a breaking change (a change in existing functionality)? YES/NO
Does this change require a documentation update? YES/NO
Does this change require an update to any of the following submodules? YES/NO (If YES, please add a link to any PRs that are pending.)
- EMC verif-global
- GDAS
- GFS-utils
- GSI
- GSI-monitor
- GSI-utils
- UFS-utils
- UFS-weather-model
- wxflow

How has this been tested?

Checklist

Any dependent changes have been merged and published
My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
My changes generate no new warnings
New and existing tests pass with my changes
I have made corresponding changes to the documentation if necessary

ush/compare_f90nml.py

As forecast ensemble jobs are added to the global workflow, this PR ensures the output is being cleaned up properly once it is no longer needed. Resolves NOAA-EMC#833

This PR updates the parm/config/gfs/config.resources and env/WCOSS2.env files for the BUFR sounding job "postsnd." It includes adjustments to resource settings such as tasks per node and memory allocations for various GFS resolutions, including C768, C1152, and others. Here are the proposed changes: C768: 7 nodes, 21 tasks per node C1152: 16 nodes, 9 tasks per node

NCO has requested that each COM variable specify whether it is an input or an output. This completes that process for the global-workflow Unified Post Processor (UPP) task. Refs: NOAA-EMC#2451

This PR updates the `develop` branch to use the newer operational `obsproc/v1.2.0` and `prepobs/v1.1.0`. The obsproc/prepobs installs in glopara space on supported platforms use tags cut from the `dev/gfsv17` branches in the respective repos. The installation of `prepobs/v1.1.0` on WCOSS2 is called "gfsv17_v1.1.0" to help avoid GFSv16 users using it instead of the operational module. Also, the `HOMEobsproc` path is updated to set an empty default for `obsproc_run_ver`. This both removes the need to set a default (and constantly update it, which is duplication) and avoid the unset variable error when the fcst jobs use their own load module script that does not know `obsproc_run_ver`: ``` export HOMEobsproc="${BASE_GIT:-}/obsproc/v${obsproc_run_ver:-}" ``` This PR also reverts the prepobs and fit2obs installs on MSU back to the glopara space from the temporary `/work/noaa/global/kfriedma/glopara` space installs. Lastly, this PR also includes updates to complete issue NOAA-EMC#2844 (merge `build.spack.ver` and `run.spack.ver`). Resolves NOAA-EMC#2291 Resolves NOAA-EMC#2840 Resolves NOAA-EMC#2844

…fire_blending

aerorahul

@bbakernoaa
Please let me know if any of my comments need clarifications

parm/config/gefs/config.prep_emissions

parm/prep/aero_emissions.yaml

ush/python/pygfs/task/aero_emissions.py

Co-authored-by: Rahul Mahajan <[email protected]>

…ng' into feature/fire_blending

scripts/exglobal_prep_emissions.py

ush/python/pygfs/task/aero_emissions.py

Co-authored-by: David Huber <[email protected]>

ush/python/pygfs/task/aero_emissions.py

aerorahul · 2024-10-18T16:40:11Z

@bbakernoaa
Others, please don't hold this PR for my review. I think I have reviewed this PR and will let others review and provide comments.

Co-authored-by: David Huber <[email protected]>

resolving comments

WalterKolczynski-NOAA

Let me know if you have questions about the refactoring to keep the data inside the class (or anything else).

parm/config/gefs/config.prep_emissions

WalterKolczynski-NOAA · 2024-10-23T09:50:00Z

ush/forecast_postdet.sh

  # link directory containing GOCART input dataset, if provided
  if [[ -n "${AERO_INPUTS_DIR}" ]]; then
    ${NLN} "${AERO_INPUTS_DIR}" "${DATA}/ExtData"
    status=$?
    [[ ${status} -ne 0 ]] && exit "${status}"
  fi

+  # Link blending emissions if AERO_EMIS_FIRE == blending
+  if [[ "${AERO_EMIS_FIRE}" == "blending" && "${RUN}" == "gefs" ]]; then 


Is there a reason for the check against $RUN? Presumably, if $AERO_EMIS_FIRE is blending, we want to use blending regardless of the $RUN.

I'm not sure... Rahul asked to add this but

Since this was only GEFS related, I suggested the RUN condition check. Pretty sure, SFS will need that as well in the future.
I can go either way.

I'd rather rely on the app config to know whether it wants blending.

ush/python/pygfs/task/aero_emissions.py

scripts/exglobal_prep_emissions.py

ush/python/pygfs/task/aero_emissions.py

Co-authored-by: Walter Kolczynski - NOAA <[email protected]>

aerorahul · 2024-11-04T19:54:45Z

ush/forecast_postdet.sh

  # link directory containing GOCART input dataset, if provided
  if [[ -n "${AERO_INPUTS_DIR}" ]]; then
    ${NLN} "${AERO_INPUTS_DIR}" "${DATA}/ExtData"
    status=$?
    [[ ${status} -ne 0 ]] && exit "${status}"
  fi

+  # Link blending emissions if AERO_EMIS_FIRE == blending
+  if [[ "${AERO_EMIS_FIRE}" == "blending" && "${RUN}" == "gefs" ]]; then 
+    ${NCP} "${COMOUT_CHEM_HISTORY}/${RUN}.${vdate:0:8}.${RUN}.blended_emissions.nc" "${DATA}"


Where is vdate coming from in this function scope?

WalterKolczynski-NOAA

Mostly minor stuff in the second pass, but there are a couple suggestions from my previous review that were marked as resolved without being addressed (I marked them back as unresolved).

WalterKolczynski-NOAA · 2024-11-05T17:19:10Z

ush/python/pygfs/task/aero_emissions.py

+        nforecast_hours = self.task_config["FHMAX_GFS"]
+        blend_start_date = self.task_config.PDY


This is my fault, I should've been consistent when I made the suggestion:

Suggested change

nforecast_hours = self.task_config["FHMAX_GFS"]

blend_start_date = self.task_config.PDY

nforecast_hours = self.task_config["FHMAX_GFS"]

blend_start_date = self.task_config["PDY"]

WalterKolczynski-NOAA · 2024-11-05T17:29:41Z

ush/forecast_postdet.sh

  # link directory containing GOCART input dataset, if provided
  if [[ -n "${AERO_INPUTS_DIR}" ]]; then
    ${NLN} "${AERO_INPUTS_DIR}" "${DATA}/ExtData"
    status=$?
    [[ ${status} -ne 0 ]] && exit "${status}"
  fi

+  # Link blending emissions if AERO_EMIS_FIRE == blending
+  if [[ "${AERO_EMIS_FIRE}" == "blending" && "${RUN}" == "gefs" ]]; then 


I'd rather rely on the app config to know whether it wants blending.

WalterKolczynski-NOAA · 2024-11-07T19:01:06Z

ush/python/pygfs/task/aero_emissions.py

+            {
+                "cdate": blend_start_date,
+                "nforecast_days": nforecast_hours // 24,
+                # "forecast_dates": forecast_dates,


Suggested change

# "forecast_dates": forecast_dates,

WalterKolczynski-NOAA · 2024-11-07T19:02:53Z

ush/python/pygfs/task/aero_emissions.py

+        --------
+        None
+        """
+        Config_dict = self.task_config['config']


Object instances should be lowercase.

Suggested change

Config_dict = self.task_config['config']

config_dict = self.task_config['config']

WalterKolczynski-NOAA · 2024-11-07T19:03:34Z

ush/python/pygfs/task/aero_emissions.py

+        workdir : Union[str, os.PathLike])
+            The working directory path.
+        current_date: (str, optional)
+            The current date for fire emissions generation.
+        forecast_dates: list, optional
+            List of forecast dates.
+        Config_dict:  Dict
+            Configuration dictionary for the task.


Not passing in any parameters anymore.

Suggested change

workdir : Union[str, os.PathLike])

The working directory path.

current_date: (str, optional)

The current date for fire emissions generation.

forecast_dates: list, optional

List of forecast dates.

Config_dict: Dict

Configuration dictionary for the task.

None

WalterKolczynski-NOAA · 2024-11-07T19:09:33Z

ush/python/pygfs/task/aero_emissions.py

+            for v in ObsEmis.data_vars:
+                if not scale_climo:
+                    if tslice > n_persist:
+                        # kk = ratio * dset[v] + (1 - ratio) * climo[v].data[tslice, :, :]


Suggested change

# kk = ratio * dset[v] + (1 - ratio) * climo[v].data[tslice, :, :]

bbakernoaa marked this pull request as ready for review September 13, 2024 13:05

bbakernoaa force-pushed the feature/fire_blending branch from ee128da to fd0e7a5 Compare September 13, 2024 13:48

bbakernoaa marked this pull request as draft September 13, 2024 14:03

add bones to the fire blending in the prep_emissions job

c8e6344

bbakernoaa force-pushed the feature/fire_blending branch from fd0e7a5 to c8e6344 Compare September 13, 2024 15:09

bbakernoaa and others added 5 commits September 13, 2024 15:19

pycodestyle

4d81cb8

pycodestyle fixes

5f86abd

fix prep_emission jjob

62dbf31

Merge branch 'develop' into feature/fire_blending

ae3f599

Merge branch 'develop' into feature/fire_blending

16e21bd

aerorahul reviewed Sep 14, 2024

View reviewed changes

ush/compare_f90nml.py Show resolved Hide resolved

WalterKolczynski-NOAA changed the title ~~[GOCART] Add Fire Blending~~ Add fire blending to GOCART Sep 17, 2024

aerorahul added CI-Gaea-Ready **CM use only** PR is ready for CI testing on Gaea and removed CI-Gaea-Ready **CM use only** PR is ready for CI testing on Gaea labels Sep 19, 2024

AntonMFernando-NOAA and others added 6 commits September 19, 2024 15:21

Cleanup job for GEFS (NOAA-EMC#2919)

b22d7bc

As forecast ensemble jobs are added to the global workflow, this PR ensures the output is being cleaned up properly once it is no longer needed. Resolves NOAA-EMC#833

Update global atmos upp job to use COMIN/COMOUT (NOAA-EMC#2867)

5b57604

NCO has requested that each COM variable specify whether it is an input or an output. This completes that process for the global-workflow Unified Post Processor (UPP) task. Refs: NOAA-EMC#2451

Merge remote-tracking branch 'remotes/upstream/develop' into feature/…

eb8bc78

…fire_blending

Merge branch 'develop' into feature/fire_blending

9a9a874

aerorahul requested changes Sep 25, 2024

View reviewed changes

aerorahul requested a review from AntonMFernando-NOAA October 1, 2024 14:54

bbakernoaa and others added 6 commits October 7, 2024 09:40

Update parm/prep/aero_emissions.yaml

3b87649

Co-authored-by: Rahul Mahajan <[email protected]>

Update parm/prep/aero_emissions.yaml

21d7bad

Co-authored-by: Rahul Mahajan <[email protected]>

Update parm/prep/aero_emissions.yaml

58e7280

Co-authored-by: Rahul Mahajan <[email protected]>

Update parm/prep/aero_emissions.yaml

f74f471

Co-authored-by: Rahul Mahajan <[email protected]>

Update ush/python/pygfs/task/aero_emissions.py

945508f

Co-authored-by: Rahul Mahajan <[email protected]>

add coarsen_scale to yaml and use in aero_emission.py

eb53925

bbakernoaa requested review from RussTreadon-NOAA, KateFriedman-NOAA, DavidHuber-NOAA and TerrenceMcGuinness-NOAA as code owners October 17, 2024 20:39

bbakernoaa requested a review from aerorahul October 17, 2024 20:40

bbakernoaa added 2 commits October 18, 2024 00:46

add with command when opening files

d44c5b9

Merge remote-tracking branch 'refs/remotes/origin/feature/fire_blendi…

5d2f493

…ng' into feature/fire_blending

DavidHuber-NOAA reviewed Oct 18, 2024

View reviewed changes

scripts/exglobal_prep_emissions.py Outdated Show resolved Hide resolved

DavidHuber-NOAA reviewed Oct 18, 2024

View reviewed changes

ush/python/pygfs/task/aero_emissions.py Outdated Show resolved Hide resolved

Update scripts/exglobal_prep_emissions.py

f204898

Co-authored-by: David Huber <[email protected]>