Add a "cache" parameter to the pipeline to easily toggle this feature in KFP between runs

[MichaelClifford]

There are instance while debugging where you want to ensure caching is being used for steps that we typically don't want caching in order to speed up testing of steps that occur lower in the pipeline. Instead of hard coding the boolean value like this, set_caching_options(False), we should add it as a variable that gets exposed as a pipeline parameter so that it can be toggled on or off by the user on the DSP dashboard.

[MichaelClifford]

Looks like this is actually currently not possible. Any function that impacts how the kube resource is defined needs to be set at IR compile time and cannot be used as a pipeline parameter. This is an issue KFP is aware of and there should be a fix sometime in the future.

[MichaelClifford]

As in intermediate measure, I will add a single variable that will be used for each .set_caching_option call. This will make toggling cache easier, but users will still have to compile and upload a new pipeline.yaml between each change.