passing information among different tasks

[uturuncoglu]

I wonder if there is a way of updating information read from the config.yaml file. In my case, workflow has a task that downloads forcing data and I need to point those data files when I am creating cdeps stream namelist file (defined by stream_data_files). This is could be also used to update other informations such as ESMF Mesh file (stream_mesh_file) etc. Maybe it could be another way to pass information between the tasks but I am not sure since current implementation of cdeps.atm_stream() does not accept any arguments to use. Do we need to extend it to accept options? or maybe dynamically updating (or adding entries) to config.yaml would be the option. Not sure since then the result config.yaml could end up a different one since it is dynamically changing along the way and it might not be the safest way and might be hard to debug in case of any issue. Anyway, any suggestion might help.

[maddenp-noaa]

Hi @uturuncoglu, thanks for starting this discussion. I have some questions:

1. Are the forcing-data filenames deterministic based on some aspects of the experiment's configuration? That is, is it possible to calculate in advance what the names will be, and use those names both to a) do the download, and b) modify the CDEPS / Coastal YAML config (I'm guessing that this is what we're talking about) with those values? I can imagine that there might be value in being able to modify the YAML and run the part of the experiment that uses those files while skipping the step that downloads the files -- maybe the files can be copied manually into place in some cases, like for debugging.
2. What language is your workflow code written in? If it's bash, you can use the uw config realize mode and action to programmatically set the filenames in the YAML config once you know what the filenames are. If you're in Python, you can use uwtools.api.config.realize() to do the same.
3. You could potentially opt to have the YAML config contain invariant, canonical names for the files, and rely on either an upstream task or a manual user action to create symbolic links. For example, in a different, imaginary application that consumes an NWP analysis file, the config might contain an item like atm_filename: atm.nc and another task (or a user) might ln -s gfs.t00z.atmanl.nc atm.nc for a specific run. That would delegate the knowledge about the actual filename to the process/person that knows it, relying on the convention that there will always be a symlink with a predictable name.

Do we need to extend it to accept options?

Probably not: The design of the UW drivers is such that they are expected to get any needed information from the config that the driver is instantiated with, which is also schema-checked for correctness.

or maybe dynamically updating (or adding entries) to config.yaml would be the option.

I think this is the right approach. Depending on your workflow, and especially whether it processes any tasks in parallel, you might need to consider making a copy of the main config and editing only the copy, perhaps in the run directory for a specific task.

Those are just some initial thoughts. We'd be happy to look at code, too, and try to help come up with good ideas.

[uturuncoglu]

Thanks for your help. Here are my comments,

1. I could fix the name of the files. I am using external Python module called Herbie and once I download data, I am subsetting them and I could rename them in a certain way (I also tried to combine them but it is slow, so I need to find a more efficient way). So, maybe YAML configuration file might have some fixed file names and variable names. Probably, I could update variable names without any issue.
2. I am trying to implement everything in Python. So, uwtools.api.config.realize() maybe the way to go if I need.
3. Yes, I agree. I think I could follow this approach. Let me try.

Thanks for great ideas. I just need to test them to see how this will work in the real workflow configuration. I'll update you once I test it.

[maddenp-noaa]

Hi @uturuncoglu. I just wanted to let you know that I merged a small PR updating the coastal.py driver for compatibility with the uwtools version 2.5.0 that we just released. The changes are trivial, but I thought I should let you know.

[uturuncoglu]

@maddenp-noaa Thanks. Let me try with that version too. I'll update you about it, if I have any issue.
BTW, is there any news about generating driver specific ufs.configure and model_configure files.