Guard Task execution via changed files

[zregvart]

The new Task in .tekton/tasks/changed-dirs.yaml produces a list of changed directories for a particular pull request as an array result. Given this result guards can be done, e.g. using CEL expressions to limit when a Task on the Pipeline needs to run.
This way we can skip expensive Tasks that are unrelated to the change done in the pull request.

Similar to work in #1188 and #524, with the distinction that the PipelineRun is executed, only potentially not in full.

[zregvart]

Marked as draft, not sure if this approach will work, need to un-draft to have the Pipeline run.

Any further ideas welcome!

[zregvart]

We don't seem to allow CEL expressions in when:

feature flag enable-cel-in-whenexpression should be set to true to use CEL

[chmeliik]

It does seem too limiting without enable-cel-in-whenexpression For many of the steps, guarding them on one specific directory is not enough :(

[chmeliik]

I was going to create a script that gets the list of changed files and evaluates some gitignore-style conditions in one step to enable skipping e2e-tests. It would avoid the result size limit problems and be more flexible than the default when (but less flexible than when with CEL)

I can put together a PR for it eventually

[zregvart]

It does seem too limiting without enable-cel-in-whenexpression For many of the steps, guarding them on one specific directory is not enough :(

I agree, the alternative, as you pointed out is to not build a generic solution but a specific one, which would mean moving the logic into the changed-dirs Task (and in the process probably renaming it to something like task-switchboard). Tedious to maintain and not nearly as nice.

I do think the first iteration could be to look at the top level directories, that might be good enough.

There could be a middle ground, e.g. provide a set of expressions (in CEL or something else) and when those evaluate as true emit one item in the result array of changed-dirs, then use when based on that value. If that makes sense. I can try to prototype this idea on this PR.

I was going to create a script that gets the list of changed files and evaluates some gitignore-style conditions in one step to enable skipping e2e-tests. It would avoid the result size limit problems and be more flexible than the default when (but less flexible than when with CEL)

I think that kind of logic should go to the e2e harness, e.g. using Skip based on the Tasks/Pipelines changed. Might be easier to maintain this in golang, with a bit of effort this could payoff greatly.

[zregvart]

This version evaluates expressions over all files in a PR using OPA/Rego. The expressions parameter determine by evaluating to true what result is omitted, the function strings.any_prefix_match comes in quite handy.

[chmeliik]

This version evaluates expressions over all files in a PR using OPA/Rego. The expressions parameter determine by evaluating to true what result is omitted, the function strings.any_prefix_match comes in quite handy.

I like this 👍

[chmeliik]

IMO all of these should also trigger on .tekton/ and hack/ - a step should run if the step definition changes (or if the script that implements the step changes).

[chmeliik]

(could also get more specific with it, but .tekton/ and hack/ would be good enough for me)

[zregvart]

Very true, added...

[chmeliik]

LGTM 🌮