Using jinja2 loops in config files?

[benjamin-cash]

I have a case where I am using uw file copy to copy a set of similar files into my run directory. E.g.,

ca_tile1.nc
ca_tile2.nc

etc. I would like to set this up as a jinja2 loop inside my config file. However, I find that if I give a config file with a jinja2 variable declaration in it to uw file copy or uw config realize, I get an error along the following lines:

yaml.scanner.ScannerError: while scanning for the next token
found character '%' that cannot start any token
  in "<unicode string>", line 1, column 2:
    {% set tiles = ['1','2','3','4',' ... 

I tried using uw template render to resolve the jinja2 loop, but it turns out template render can't resolve referencing values from within the yaml file, so that constructions like:

exp:
  base_dir: "{{ input_base }}/exp"
  files_to_copy:
    data_table: "{{ exp.base_dir }}/data_table"

will flag exp as an uprovided value.

Is there a way to make use of both jinja2 loops AND referencing yaml values from within the file? Or does it need to be one or the other?

[maddenp-noaa]

Can you show us a simple-as-possible reproducer?

[benjamin-cash]

{% set months = ['01','02','03','04','05','06','07','08','09','10','11','12'] %}

input_base: "/work2/02441/bcash/frontera/ufs_input"

fix:
  base_dir: "{{ input_base }}/fix"
  aer: "{{ fix.base_dir }}/aer/20220805"

  files_to_copy:
   {% for mn in months %}
    aeroclim.m{{ mn }}.nc: "{{ fix.aer }}/merra2.aerclim.2003-2014.m{{ mn }}.nc"
   {% endfor %}

In my tests this file will fail with uw template render, uw file copy, and uw config realize.

[benjamin-cash]

The reason I want to use a loop here is primarily to avoid making a mistake in matching mn on the right and left side of the copy, because it would be so easy to do. Some of these lists get very long - one of the sets of files required is a set of co2 text files that go from 1956-2022, each in its own separate file.

[christinaholtNOAA]

I like where you're going here. I've also been feeling the pain of repeating a bunch of similar file names. I'd like to see that we could maybe use some very basic globbing/pattern matching to help with DRYing out info.

I am also curious if a uwtools-supported !list pyyaml tag would help with the list expansion you're trying for in the first example.

[benjamin-cash]

Not only is it tedious, but it is a prime opportunity to introduce an error.

[benjamin-cash]

As for the !list option, I'm not familiar enough with uwtools and yaml to know what that would look like, but I'm pretty agnostic about the exact details of the solution. The main thing I'm looking for is to be able to do this from the CLI (obviously solving this via the API is possible) and to have it fairly clear from the .yaml what is going on when I go back to look at it later or hand the file off to someone.

[maddenp-noaa]

$ cat ben.yaml 
{%- set months = ['01','02','03','04','05','06','07','08','09','10','11','12'] -%}
input_base: '/work2/02441/bcash/frontera/ufs_input'
fix:
  {%- raw %}
  base_dir: '{{ input_base }}/fix'
  aer: '{{ fix.base_dir }}/aer/20220805'
  {% endraw -%}
  files_to_copy:
   {% for mn in months -%}
   aeroclim.m{{ mn }}.nc: '{% raw %}{{ fix.aer }}{% endraw %}/merra2.aerclim.2003-2014.m{{ mn }}.nc'
   {% endfor %}

$ cat ben.yaml | uw template render | uw config realize --input-format yaml --output-format yaml
input_base: /work2/02441/bcash/frontera/ufs_input
fix:
  base_dir: /work2/02441/bcash/frontera/ufs_input/fix
  aer: /work2/02441/bcash/frontera/ufs_input/fix/aer/20220805
  files_to_copy:
    aeroclim.m01.nc: /work2/02441/bcash/frontera/ufs_input/fix/aer/20220805/merra2.aerclim.2003-2014.m01.nc
    aeroclim.m02.nc: /work2/02441/bcash/frontera/ufs_input/fix/aer/20220805/merra2.aerclim.2003-2014.m02.nc
    aeroclim.m03.nc: /work2/02441/bcash/frontera/ufs_input/fix/aer/20220805/merra2.aerclim.2003-2014.m03.nc
    aeroclim.m04.nc: /work2/02441/bcash/frontera/ufs_input/fix/aer/20220805/merra2.aerclim.2003-2014.m04.nc
    aeroclim.m05.nc: /work2/02441/bcash/frontera/ufs_input/fix/aer/20220805/merra2.aerclim.2003-2014.m05.nc
    aeroclim.m06.nc: /work2/02441/bcash/frontera/ufs_input/fix/aer/20220805/merra2.aerclim.2003-2014.m06.nc
    aeroclim.m07.nc: /work2/02441/bcash/frontera/ufs_input/fix/aer/20220805/merra2.aerclim.2003-2014.m07.nc
    aeroclim.m08.nc: /work2/02441/bcash/frontera/ufs_input/fix/aer/20220805/merra2.aerclim.2003-2014.m08.nc
    aeroclim.m09.nc: /work2/02441/bcash/frontera/ufs_input/fix/aer/20220805/merra2.aerclim.2003-2014.m09.nc
    aeroclim.m10.nc: /work2/02441/bcash/frontera/ufs_input/fix/aer/20220805/merra2.aerclim.2003-2014.m10.nc
    aeroclim.m11.nc: /work2/02441/bcash/frontera/ufs_input/fix/aer/20220805/merra2.aerclim.2003-2014.m11.nc
    aeroclim.m12.nc: /work2/02441/bcash/frontera/ufs_input/fix/aer/20220805/merra2.aerclim.2003-2014.m12.nc

Is that it?

[benjamin-cash]

Yes, that looks like what I was aiming for. :) I hadn't come across the raw functionality in jinja2 before, that's very handy. If I understand correctly, it is exempting those blocks from interpretation by template render, which allows for expansion of the loop before being ingested by by config realize. Do I have that right?

[maddenp-noaa]

Exactly: Hide from template render the bits that need to be acted on by config realize.

[christinaholtNOAA]

Fancy! I don't hate this.

[benjamin-cash]

@maddenp-noaa - why the {%- -%} around months? In my limited understanding that construction is to suppress random newlines and whitespaces, but it doesn't seem like that should ever matter for that declaration. Is it just there for neatness or will there be an issue with leaving it out?

[maddenp-noaa]

It's just to keep the output tidy by avoiding creation of useless whitespace. It's not a correctness concern.

But: Humans are great at visual pattern matching, and anything that looks unintentional will likely be considered a possible issue during debugging when something has gone wrong. Especially in the case of variable / expression expansion, whitespace can easily indicate a variable that had no value. So, I never underestimate the value of output looking intentional, which usually implies tidiness. No red herrings.