An SRW User Story

[christinaholtNOAA]

I assume I’ve gained a reputation for being the uwtools pusher in most circles that overlap with me in any number of UFS and NOAA contexts. Before I was the UW Team Product Owner, I was primarily an application developer (Short Range Weather, or SRW, to be specific). Before that, I supported HWRF developers; before that, I was a grad student – and a user of scripts from the grad student before me. At each of these stages in my career, I was burdened with software that left me with the same thought: “There has to be a better way!” I started to see patterns in these various tools and applications, where some snippets of code needed for one task had been borrowed verbatim from other scripts. Sometimes this code was yanked roughly from other scripting languages, neglecting superior features of the new language. The whole enterprise seemed based on copypasta of ad hoc scripts that were somehow doing the things that the atmospheric scientists wanted but were often still enigmatic. To this day, I will get that old mysterious feeling when working with the newest generation of apps, often asking myself the question, “Why/how did this even work?” The Unified Workflow project was born out of several of us having similar feelings over large portions of our careers.

Now uwtools is on the scene, and the UW Team’s top short-term goal is to integrate it fully into the SRW App. The tables have turned, and our team will become uwtools users as much as we have been developers. To aid that transition, I have been introducing the rest of the UW Team to SRW in a way that will help them take on more of the work in the coming weeks. We started with standalone UPP: It’s a pretty straightforward part of the SRW workflow in the grand scheme of things, and so represented a good starting point.

All the information you need to run a given uwtools driver is described in one schema-checked YAML file passed to the driver as an input. The team and I sat down to fill out this UW YAML file for the new UPP driver. We realized that, by the time we’d gathered six pieces of information for the four YAML entries we were working on, we’d opened up maybe eight or nine files to cobble together the complete information necessary for running UPP. This process alone – consolidating information scattered across several files in several locations into a single well-controlled input file – gave me tremendous hope about the future of uwtools-based apps! For years, or even decades, the development of NWP applications has been conducted in a rather piecemeal fashion, largely collaborative but with no good rules or guidelines as to when and where variables should be defined, files should be linked/copied/created, or logical branches followed. This has led to the vital information for running even one component of a complex system being spread over the entirety of the scripting codebase used to configure and run an application. By congealing it into one location, I believe we’re going to flatten the NWP learning curve. By storing app-relevant information in a library of configuration files for users to start with, I am confident we will start to see much quicker wins come from the UFS Community. The team and I eventually tracked down all the information needed to run UPP and were confident that it was enough to start testing the app with the UW UPP driver.

Don’t get me wrong: I’ve believed in the UW project since its inception, and I’ve used uwtools for nearly every NWP side project I’ve been involved in. I have also pushed the package on nearly every colleague who throws me a problem it can solve, but the thing that came next with the uwtools – SRW integration – was what completely sold me.

I was modifying the SRW ush/config_defaults.yaml file in many ways. I replaced the old UPP section with the new UW YAML information and found that I would need to load the updated config_defaults.yaml file with the UW config tool. I’ll save you the gory details; let’s just say that parts of the SRW YAML file were not compatible with the very generic uw config tool, so other modifications were in order. There were a lot of changes in the end.

The standard way to test these changes is to run a single SRW Workflow End-to-End (WE2E) test to create the experiment directory – complete with its necessary configured files – and try to run the experiment. When things fail, fix them and try again. With such a large set of invasive changes, the likelihood of things failing on these modifications was quite high.

For the first time, I had the power to check this heavily modified ush/config_defaults.yaml for compatibility and correctness issues without ever trying to define and generate an experiment: I used the uw config tool! Here’s the case: I had a YAML file with some Jinja2 expressions peppered in, and I knew I wanted them to be mostly replaced by the text they were describing – a process we call dereferencing. (A handful of exceptions can’t be dereferenced until run time.) So I ran the defaults file through the uw config realize tool (it not only does the dereferencing step, but also allows a user to optionally apply modifications from other config files), and iterated through syntax errors and references that should have been rendered until it gave me something that looked like what I wanted – a ready-to-use YAML experiment configuration file. Not only was I just realizing this default file over and over again to fix the lines I’d botched up (making sure integers were really integers and paths looked like paths), but I also had the --values-needed flag at my disposal. Total game changer! It gave me a concise summary of all the lines left unrendered. I could then determine whether I expected a rendered line, or if it was a value that run-time scripts and/or drivers are responsible for. The use of uwtools let me fail so much earlier in the process without wasting HPC resources (or my time!) doing everything I didn’t need to do to debug this single configuration file. I was in love, to say the least.

Our UW Team is still elbow-deep in this first UW driver integration, and I am confident that other success stories will emerge from the process to show us the answer to the age-old question I’ve had since grad school – there really is a better way!