Import Rocoto workflow and execute it

[christopherwharrop-noaa]

Chiltepin currently has a very limited capability and users have no way to run complex NWP workflows with Parsl and/or Globus Compute in a federated fashion. While native pure Python capabilities are coming to Chiltepin, users must build those workflows from scratch and do not have any way to reliably construct production quality workflows currently available in SRW or other UFS Apps.

To bridge the gap between the UFS workflows currently available and their federated counterparts, a translation layer is needed so that a Rocoto workflow can be imported into Chiltepin and executed using Parsl and/or Globus Compute.

The workflow generation process for the UFS (and other) Apps is complex, and it is not feasible to integrate Chiltepin into that procedure to generate Chiltepin workflow code at UFS App workflow creation time. Instead, we can build tools that allow Chiltepin to import a pre-made Rocoto XML file, create the necessary Parsl Apps or Globus Compute functions from it, and then launch the workflow on the appropriate resources. If this was implemented, it would mean that Chiltepin could run any Rocoto workflow!

To do this, one could parse the Rocoto XML directly using xml.etree.ElementTree and create the necessary Chiltepin objects to represent the workflow tasks, cycles, and dependencies. Or it might be easier to convert the XML to a dict using xmltodict and then process the workflow as YAML to create the necessary objects.

The most difficult aspect of this work is handling the dependencies. With Parsl, dependencies are injected by passing Futures when calling the "Apps". But, in order to do that, you have to know what that task depends on when calling it. The way Rocoto's XML is designed, tasks know what they depend on, but they do not know what tasks depend on them. So, that information must somehow be made available at calling time.

Another issue is that Parsl only has the ability to handle "and" dependencies. If a list of Futures is passed when calling an "App" that App will not run until the Future is completed successfully. However, if you want to trigger an App if (A or B) is true, then you must first construct a Future that tests that condition, and pass that future. This would/could be handled by Parsl's Join App. Join Apps must be used to construct complex dependency expressions.