Retrigger step does not keep context

[magnars]

Thanks for making a cool project. LambdaCD is just what I was looking for.

When retriggering a step in a build, a new build is started from that point. However, since the context changes, the files generated by earlier steps are missing.

This might not be a bug, but it did break my expectations. :-)

I would expect the old context to be reused in the new build, so that I could try a broken build again.

What am I missing?

[flosell]

Thanks for the feedback! You hit one of the rough spots where I haven't figured out a good solution yet (and I'd be happy for ideas).

To answer your question first:
If I understand you correctly (please correct me if I'm wrong) your scenario is similar to the one described in #36 so maybe the solution introduced there (marking some steps so they can't be retriggered) can at least resolve some confusion when using the pipeline. There's a bit more on this deep in the documentation.
On top of that, you might be able to work around the problem by changing your pipeline or approach: Retrigger the parent of the step (e.g. workspace) so that all your steps (including the one that generates the files) are retriggered. Put the files you need on an artifact store and pull them back whenever you need them (maybe lambdacd-artifacts is good enough if you don't have/want a proper artifact store.

But your question also prompted me to think about this problem again after a while so maybe there's a way to improve LambdaCD to support this better:

• There's no inherent reason why with-workspace needs to throw away all data at the end of the step. A workspace that keeps data around and re-uses it on retrigger could be added (we already have the information which build was retriggered in the ctx). If I remember correctly, the original decision was made so that we don't have to deal with housekeeping for old builds and because it felt more "pure" from a Continuous Delivery standpoint: A retriggered build that starts off from an arbitrary state of a previous build might behave differently, leading to less predictable outcome (e.g. files from a subsequent test stage that end up in a release artifact).

• It's hard to come up with a general retriggering behaviour for LambdaCD pipelines because they (being code) can have arbitrary behaviour and structure. But we can declare which strategy is being used for a container step (using the :retrigger-predicate on execute-steps) so we might be able to build on this to become smarter about what needs to be retriggered.

[magnars]

Thanks for your in-depth answer! Yes, my scenario does seem like the same that is described in #36. I think the point about steps running arbitrary code illustrates well why this is a hard problem.

To be clear what my use case was:

• I had just fixed the definition of a step while developing.
• This step needed some clojurescript files to be compiled first. They take a minute to run.
• I tried retriggering to avoid the recompilation to save a little time.

So running from an earlier step wouldn't help me. And it wasn't a big deal either. But my intuition about how the retriggering would work was a bit off. :)

I am a bit curious: What use cases you have for the retriggering the way it works now?

[crimeminister]

I have found that adding an immutable fileset to a pipeline can improve things with respect to build output. There's still housekeeping to do w.r.t. artifacts but using filesets makes it easy to do so in a functionally idiomatic way.

[flosell]

@crimeminister filesets look really interesting, thanks for the suggestion! Do you have an example what that looks like in an actual pipeline?

@magnars I have seen a lot of people write pipelines where a single step runs unit-tests and builds a complete artifact (jar, Docker image) and then has subsequent steps to deploy to environments and run additional tests against those. There, retriggering isn't too much of a problem since the steps are more self contained (and slower).

But I'd actually prefer your approach of more fine grained steps that might depend on each other, both for speed of development and visibility so I'll definitely keep this issue open to find a way how LambdaCD can support this better.

[magnars]

Thanks for the clarification, that makes sense. I'm glad to hear that you want to support smaller steps as well. :)