Fully customizable pipeline definitions

[KuraiAndras]

I have encountered a number of projects where I had to customize pipelines in ways that the Nuke pipeline attributes could not handle. My usual solution would be to either manually write the pipeline or do some hacky workaround to generate a customized pipeline file.

I have been thinking and came up with a solution I would like to pursue, and I would like to get some critique before pushing forward (If I finish this then it will be open sourced). Keep in mind that I have mostly used nuke with GitHub Workflows and Azure Pipelines.

I like nuke's generated pipelines, but they have a few problems:

• No support for running on self-hosted runners
• No support for running inside containers
• Built-in steps might be behind on versions
• Sometimes the easiest and most sane solution is to have steps before you run nuke (my case was I needed to install cloudsmith)

So, I thought about making an alternative which gives you full control of the generated files.

The sample build we want to run:

partial class Build : NukeBuild
{
    public static int Main() => Execute<Build>(x => x.Run);

    [Parameter("Configuration to build - Default is 'Debug' (local) or 'Release' (server)")]
    readonly Configuration Configuration = IsLocalBuild ? Configuration.Debug : Configuration.Release;

    Target SayHello => _ => _
        .Executes(() => Log.Information("Saying Hello"));

    Target Run => _ => _
        .Executes(() => Log.Information("Run completed, configuration is {Configuration}", Configuration));
}

Current generation parameters:

[GitHubActions
(
    "ci",
    GitHubActionsImage.UbuntuLatest,
    FetchDepth = 0,
    OnPushBranches = new[] { "main" },
    CacheExcludePatterns = new[] { "*.tar" },
    InvokedTargets = new[] { nameof(Run), nameof(SayHello) }
)]

My approach:

// We specify a factory type so we are not restricted to primitives
[GithubActionsExtended(typeof(Factory))]
partial class Build
{
    // Factpry is nested type, so we don't need to specify Build.SayHello and others
    public class Factory : IGithubActionsImplementationFactory
    {
        // Using the new target-typed new expressions helps a lot but it is a bit too verbose
        // Might be improved by terse formatting?
        public GithubWorkflowDefinition Create() => new()
        {
            FileName = "extended",
            Name = "extended",
            On = new()
            {
                Push = new()
                {
                    Branches = new[]
                    {
                        "main"
                    }
                }
            },
            Jobs = new()
            {
                { "ubuntu-latest", new()
                    {
                        Name = "ubuntu-latest",
                        RunsOn = "ubuntu-latest",
                        // Fully custom steps
                        Steps = new Step[]
                        {
                            new Checkout() { FetchDepth = 0 }, // Custom checkout step
                            new Cache() { CacheExcludePatterns = new[]{ "*.tar" } }, // Custom cache step
                            new Step(){ Run = "echo \"After\"" }, // Generun step
                            new NukeRun() { InvokedTargets = new[]{ nameof(Run), nameof(SayHello) } }, // cusrom step for running nuke
                            new Step(){ Run = "echo \"Before\"" }
                        },
                    }
                }
            }
        };
    }
}

Right now, this will produce the following output:

# ------------------------------------------------------------------------------
# <auto-generated>
#
#     This code was generated.
#
#     - To turn off auto-generation set:
#
#         [GithubActionsExtended (AutoGenerate = false)]
#
#     - To trigger manual generation invoke:
#
#         nuke --generate-configuration GitHubActions --host GitHubActions
#
# </auto-generated>
# ------------------------------------------------------------------------------

name: extended
on:
  push:
    branches:
      - main
jobs:
  ubuntu-latest:
    name: ubuntu-latest
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0
      - name: 'Cache: .nuke/temp, ~/.nuget/packages'
        uses: actions/cache@v3
        with:
          path: >-
            .nuke/temp

            ~/.nuget/packages

            !*.tar
          key: ${{ runner.os }}-${{ hashFiles('**/global.json', '**/*.csproj', '**/Directory.Packages.props') }}
      - run: echo "After"
      - name: 'Run: Run SayHello'
        run: ./build.cmd Run SayHello
      - run: echo "Before"

Stuff that's missing but might be possible:

• Automatically handle targets with .Produces()
• Automatically import secrets from .Requires()
• Add all missing properties from GitHub 's documentation
• Supporting other providers - Azure, Jenkins, AppVeyor, etc...

So how does this work?
The basic idea is that surely GitHub workflows can be represented with simple POCOs which are then serialized to YAML. So the GithubWorkflowDefinition created by the factory will be serialized using tools provided by nuke's ConfigurationAttributeBase

some code from the implementation:

public class GithubWorkflowDefinition
{
    [YamlIgnore]
    public required string FileName { get; init; }

    public required string Name { get; init; }
    public string? RunName { get; init; }

    public required On On { get; init; }

    public required Dictionary<string, Job> Jobs { get; init; }
}
public class Step
{
    [YamlMember(Order = int.MinValue)]
    public virtual string? Name { get; set; }
    [YamlMember(Order = int.MinValue + 1)]
    public virtual string? Uses { get; set; }
    [YamlMember(Order = int.MinValue + 2)]
    public virtual string? Run { get; set; }
    [YamlMember(Order = int.MinValue + 3)]
    public virtual Dictionary<string, object>? With { get; set; }
}
public class Checkout : Step
{
    [YamlIgnore]
    public string Version { get; init; } = "3";

    [YamlIgnore]
    public int? FetchDepth { get; init; }

    public override string? Uses => $"actions/checkout@v{Version}";

    public override Dictionary<string, object>? With => FetchDepth is null
        ? null
        : new()
        {
            { "fetch-depth", FetchDepth },
        };
}
public class NukeRun : Step
{
    [YamlIgnore]
    public required string[] InvokedTargets { get; init; }

    [YamlIgnore]
    public string RunCommand { get; init; } = "./build.cmd";

    public override string? Name => $"Run: {string.Join(' ', InvokedTargets)}";

    public override string Run => $"{RunCommand} {string.Join(' ', InvokedTargets)}";
}

Stuff that is not in scope:

• Making it 100% bulletproof. I do not plan on making this in a way which prohibits the user from making a faulty pipeline definition
• Making it terser/more beginner friendly than nuke's current implementation - if you like using the current way of generating pipeline definitions then keep using them. This would be for scenarios where more customization is needed, but you still want nuke to do some heavy lifting

I hope some of you might like this idea, so if you have some feedback, please share (even if you don't like this idea!)

[BusHero]

Why use a C# DSL to generate a pipeline instead of writing the pipeline manually?

[matkoch]

nuke already has all the information about build dependencies.. it makes perfect sense to generate the yaml from that single source of truth..

[BusHero]

It makes sense in the case of simpler pipelines, not so sure if it scales to more complex scenarios. These scenarios may involve:

• Custom steps
• Multi job pipeline
• Vendor dependent features
• Multi vendor pipelines.

Let's say nuke does end up generating pipelines for all of these exotic scenarios, how much configurability will nuke allow? Will you have the full control over the generated pipeline? If your goal is maximum configurability, you will end up with a C# DSL that resembles pipeline's YAML dialect. Does this provide more benefits than writing the pipeline directly? I'm doubtful about.

[matkoch]

I'm not doubtful. it's just a challenge that has to be accepted by someone.

[Danielku15]

I also wanted to add some bits to this dicussion. I also decided to move to Nuke for one of my new OSS projects and we likely also want to adopt it in our company for most of our pipelines written in a variety of languages (powershell, batch, msbuild,..). The post is very long but might give some good insight into what I am thinking.

tl;dr: How about, as a first improvement step, we adopt the "GitHub Actions" concept of an "Action" into Nuke? As a "unit" which has inputs, performs steps and produces outputs it can be reused and integrated into Targets. Those special "Actions" can be called multiple times, are producing special build logs (grouping, time measurement etc.) and might even be shown in the execution plans.

Long version follows here

The problem

Task/Target based runners have definitly their place and fulfill a nice range of usecases but they suffer often of similar pain points when used in larger workflows:

• Reusing tasks with multiple inputs is often not easily achievable. e.g. how would a matrix build in Nuke look like? Or in MsBuild you might use batching but multiple CallTargets also fails to serve this need.
• Having conditional branching is often a pain. You end up with tons of conditions each on individual task level do ensure that in all the different pipelines a task might be used, it is considered correctly.
• Controlling the right order of parallel/sequential execution is often tricky.

Here a screenshot of the project I work currently on. This is "only" the GitHub Actions level jobs:

In the jobs I often have a Matrix strategy calling parameterized a single target. There are also additional inputs from outside. like whether some cached files should be used or not.

The entrypoint is often a target with certain parameters which then has dependent targets being executed. It took me really long time and many tradeoffs to make certain things reusable. I have plenty of OnlyWhenStatic, DependsOn and Before mixing targets from different pipelines in the shared ones. Yes it works, but it's not really nice.

Also my pipeline is rather encoded in GitHub Actions yamls and executing the same range of builds locally is not possible out-of-the-box.

For many developers it might be a lot more natural to develop conditions, loops and call methods than working on a task list and dependencies. Yes, you can do that today, but you will loose a lot of Nuke capabilities like nice outputs, time measurement, execution plan visualization etc.

Brainstorming a solution

I haven't thought through how such a pipeline syntax in Nuke could really look like but it is anyhow more important first to find out what conceptual approach Nuke should follow. I think it could be based on various aspects of GitHub Actions:

• A new standalone building block named NukeAction defines a certain unit of work. The idea of them is to be reusable and sharable (e.g. as NuGet packages). A separate construct avoids confusion and ensures only available features are listed (e.g. actions wouldn't have .DependsOn, .After, .Before etc). They have some similarities with the Built-In CLI Tools with Fluent APIs but follow a bit a different mindset.
• NukeActions when used as "steps" should produce log outputs similar to targets with log grouping in CI systems supporting, time measurement etc. This is important just like for target sequences to see how the pipeline is progressing and executing.
• NukeActions can be called from other NukeActions, in Targets as dependency, in Targets manually in the execution function and also in "Jobs". This way they can be a benefit to classical Target approaches and newer pipeline approaches.
• NukeJobs are consisting of steps calling predefined NukeActions or inline actions. For each step there are conditionals whether to execute the step. They fulfill the role of certain "stages" in the overall pipeline.
• NukeWorkflows are consisting of NukeJobs put into order (sequential, parallel) and the execution paths might be enriched with conditions. The workflows are putting all the things together into final pipelines.

Today we have "only" Targets which are somehow a mix between Jobs and Workflows.

The concepts would compare like this:

Nuke: NukeAction

• GHA: Actions
• TeamCity: Steps
• Azure DevOps: Tasks
• Jenkins: Steps

Nuke: NukeJob

• GHA: Jobs
• TeamCity: Build Configuration
• Azure DevOps: Job
• Jenkins: Job

Nuke: NukeWorkflow:

• GHA: Workflows
• TeamCity: Maybe Sub Projects or Composite Build Configurations?
• Azure DevOps: Pipelines
• Jenkins: (Multi-branch) Pipelines

Such a structure should then also allow proper code generation where the Nuke build.exe is ultimately called with the Job name. Locally the whole workflow could also be executed.

Conclusion

CI/CD systems are focusing on very similar concepts of a "Pipelines > Jobs > Steps" hierarchy which is controlled through further conditions. Nuke has some overlaps on the Steps/Jobs areas but does not fully match into those principles with primarily Targets as first-class citizen.

Adopting various principles into Nuke could be a big benefit as devs and devops' dont need to change their mindset when using Nuke but only need to know the terminology. I personally think it would fit into the principles and goals of Nuke quite well to also adopt those aspects.

The adoptions could be done stepwise with adding NukeActions as a start to have defined and reusable building blocks. Later job/workflow additions could be made as own construct with execution plans, conditionals, matrix builds etc. With actions available people might be able to build the pipelines already in the way they like and we don't even need to add everything.

References:

• https://blog.jetbrains.com/teamcity/2019/04/configuration-as-code-part-4-extending-the-teamcity-dsl/#defining-the-pipeline-in-code
• https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsif
• https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstrategymatrix
• https://www.jenkins.io/doc/book/pipeline/syntax/

[Tiberriver256]

This is pretty sweet! I was hunting for the same thing for a slightly different reason.

There are a number of Azure Pipelines features that cannot be executed locally and are only provided via Azure Pipelines YAML.

A couple examples:

• The VSTest task - The task itself is pretty tightly coupled with Azure Pipelines in order to provide the parallelization of a single set of tests across Azure DevOps build agents (i.e., I want to run 30 tests divvied up among 3 agents).
• Job Parallelization - In order to run the same job across several operating systems I would have to do this in YAML.

One thought is that you could potentially auto-generate your POCO classes instead of manually keeping them up to date by tapping into the JSON schema files Github and Azure DevOps use to validate their YAML.

High level steps

Step 1 - Download the schema

The JSON Schema can be downloaded from ADO REST API or a variety of locations for GitHub

• ADO - Gives you all schemas for the main yaml syntax and all extensions available within the given organization
• GitHub
  • Workflow
  • Actions - POCO generation for individual actions would involve downloading the action.yaml file for each action and using that to generate code.

Step 2 - Generate the POCO classes

1. Use NJsonSchema.CodeGeneration for anything that's in a JSON Schema
2. Write a custom generator for making POCOs from action.yaml files which are pretty simple.

Step 3 - Serialize the classes to YAML

I'm assuming something like YAMLDotNet would fit in good here

Some cons to this solution

• No expression support - Both Github workflows and ADO pipelines have a special expression syntax that doesn't conform to general YAML rules and is generally the bane of everyone's existence... My thought is that the expression syntax is there typically to either reference parameters/secrets at the task level which you guys already came up with a cool solution for or it's there to avoid repeating YAML over and over in order to keep things consistent which I think is less of problem if your YAML is auto-generated anyway.
• Pretty big effort - Seems like a pretty big lift to just get the POCO objects. If they already existed as nuget packages maybe it wouldn't be too bad to plug them in here but generating those classes is probably too big of a lift to just merge into this repo.

Suggestion

Maybe start a new repo that's just dedicated to auto-generating the POCO classes and publishing them as nuget packages and then once that work is done and on some sort of a schedule to ensure it's always up to date then look at leveraging them within this project.

[MikDal002]

I think that every possibility to extend Nuke to own specific need would be great. Actually making sth just a little more complex requires coping hundreds of lines from repository to own code just to provide small functionality. Enough to mention the own Agent Name, which isn't uncommon in professional scenarios. I think it would be great to see some kind of road map for better fulfilling Open-Close Principle - opened to extend, closed to changes.
One of important thing for me is easier debug of pipeline files generation. At this moment, this part is a pain, causing limited discoverability of possible features. Interesting would be a possibility to create whole pipeline from scratch in user code. Actually, many important parts are behind the wall of internal access modifier.

[matkoch]

I did not read everything, but I think I understand what problem you're trying to solve. Attributes for YAML generation were a pretty novel approach, but they also have the shortcomings of constants-only + no OOP. I'm imagining something like this (feat/expoando branch):

    GitHubWorkflow Base = new GitHubWorkflow()
        .Checkout(fetchDepth: 0)
        .EnableGitHubToken()
        .PublishArtifacts(false)
        .CustomStep($"""
            - task: xxx
              with:
                foo: {SomeDirectory}
            """)
        .Invoke((ITest x) => x.Test, (IPack x) => x.Pack);

    GitHubWorkflow WindowsLatest => Base
        .Name("windows-latest")
        .OnPush([MasterBranch, $"{ReleaseBranchPrefix}/*"])
        .OnPullRequest([DevelopBranch])
        .Image(GitHubActionsImage.WindowsLatest);

The first step would be to gather enough examples for best use-case coverage. I suggested that in Slack a while back already, but no one raised their hand. If there's interest here, let me know.

[KuraiAndras]

Yes, sadly I never got to fully implement this. I used an attribute to hook into nuke's file generation.

The property + fluent API sounds awesome, but it might get complicated. I chose the oo approach to mimic a yaml file's structure

[matkoch]

I don't think it's too complicated. The most complicated part is the API design, which is why a reasonable size of examples is necessary.