CLI: cache outputs & Fuzzy Starts

[josephjclark]

Short Description

This PR:

• Writes the output of each step/job to disk if --cache-steps is passed
• Uses cache data as input when running with --start if input is not explicitly passed in
• Supports "fuzzy" matching on start node names (ie, pass a partial node id or name and it'll match)
• Supports --end and --only as CLI arguments

Related issue

Fixes #375

Docs

PR open at OpenFn/docs#467

Caching

If the --cache-steps flag is passed to the CLI, every step will write its output to a local folder as a JSON file.

Caching is useful for debugging, but the real benefit comes from re-running a workflow from a fixed point.

If the --start flag is passed, the CLI will automatically load the correct input state from the cache for that step. This is is clearly logged.

Note that --cache-steps and --start are mutually exclusive. If you run --start without --cache-steps then the cached output will NOT be updated.

To work out the right input state, the CLI must find upstream step of the start job. At the moment, this is easy because in a workflow, each step can only have one input. If that rule ever changes, this part of the feature becomes more complex (I've added a comment in workflow validation to this effect).

The cache is written to a folder called .cli-cache adjacent to the workflow or job file. A sub folder will be created with the workflow name (which defaults to the file name), and a json file will be created for each step (with the step id).

So for .tmp/workflow.json you'll get a cache path something like ./.cli-cache/workflow/step-1.json.

Caching is off by default.

Setting the OPENFN_ALWAYS_CACHE_STEPS env var will default step caching ON. Disable by passing --no-cache-steps.

Fuzzy Starts

The PR also enables "fuzzy" start points to be defined.

• If the start node exactly matches a step id, that step will be used as the start
• If the start node partially matches a step id OR NAME, that step will be used
• If the start fuzzily matches multiple steps, an error will be thrown.

This behaviour exists outside of the caching stuff, but the caching stuff benefits from it because it's easier to specify a start node now.

The idea here is that a) you may want to use the step name or id as the start point, depending on what your workflow json looks like; and b) if you're using a project downloaded from Lightning, the ids are a nightmare to type

TODO

• Tidy typings
• Add a bunch of unit and integration tests
• Use the logger properly in cli/src/util/cache.ts
• Add corresponding docs (readme and docs repo)
• Maybe add an env var to default cache enabled (for power users)
• Add an API to clear the cache I am not going to do this, its actually easier for everyone to rm -rf ./cli-cache
• Consider what happens in the event of an error
• Consider what happens if state is falsy (does it even matter?)
• See "name or id?" - I think basically we need to allow --start to fuzzy match against step names and ids, for good UX
• Ensure the worker will always run with the cache off (this is trivial if cache is off by default, or else may need a commit on the worker)
• We should probably clear the cache on every run (except the start step), so that renaming steps doesn't mess things up, or older steps which didn't run don't interfere with the output.
• generate a .gitignore?
• Make --cache and --start mutually exclusive (probably)

Checklist before requesting a review

• I have performed a self-review of my code
• I have added unit tests
• Changesets have been added (if there are production code changes)

[josephjclark]

I am not going to implement a clear cache command. There's too much low level complexity associated with it.

If you do openfn clear-cache, it's not clear which cache. The repo cache? The metadata cache? The job cache? It's no good.

We could take a path to the cache. But is that a path to .cli-cache? The parent folder? The target workflow? Do you want to clear for all workflows or just for one workflow? Basically the path feels so ambiguous and unintuitive, it's not clear to me what it'll do.

Ok, so we can confirm the path before we do anything, but it's still an annoying command. Isn't rm -rf just easier
anyway? We all know what that does.

The other option is openfn workflow.json --clear-cache, which will clear the cache associated with that workflow. But it's weird because it won't run the workflow. It's even weirder if you do the long form of openfn execute workflow.json --clear-cache. It also doesn't help you clear the cache for all workflows.

Maybe we'll come back to it later - right now it feels super high complexity for incredibly low value.

[josephjclark]

Hi @mtuchi, when you've got a spare half hour can you take another look at this, maybe run a couple of tests? Thanks!

[mtuchi]

@josephjclark i have tested the --cache-steps and --start step-id. Everything work as expected
I have also tested openfn workflow.json --clear-cache. It didn't work, I don't think we need it honestly. It's very easy to clear .cli-cache if i needed

[josephjclark]

Hi @mtuchi - last one I hope!

This PR now supports --end and --only

I plan to merge and release this in the morning. Let me know if you get a chance to test it out

[josephjclark]

This interaction here - not just the working out which step to run, but how it is fed to the runtime AND reported to the user - is not well tested.

It's taken quite a while to work out a good UX.

I don't want to spend much more time on this but maybe I'll see if I can find a good way to test this. I kinda worry that it'll take half a day to get a good suite of tests on it - not worth it right now.