Support configuration templates

[tomzo]

Templates in configuration repositories

In my understanding GoCD pipeline templates were introduced at all is because any organization faces some similarity in the pipelines they declare. User can have a template with some configuration pre-defined and parameterized. So that rather than declare a full pipeline with all its stages, jobs and tasks, one may provide a few key-value pairs and tell GoCD to expand a template. This is basically a feature to support the DRY (Don't repeat yourself) priciple.
Internally this is exactly what happens, XML parameters are replaced by some variables which are in the scope of particular pipeline definition, eventually pipeline configuration instance expanded from template is not different in any way from explictly defined one. It is also not any different than a pipeline returned by configrepo extension point to server.

Once in a while a GoCD user will come along and say that he has found a pattern in all his pipelines, but it is impossible to express it with current pipeline-template model. A few examples:

• I want pipeline name to be computed from one or more parameters.
• I want all pipeline parameters to be computed from the sole SCM repository address.
• I just want one stage/job/task to be exactly like some-url but to manually configure the other stages/jobs/tasks.
• I want the parameter in my template to be a list or object, not just a key-value string.
• I want to expand these 4 complex parameters into 8 pipelines

I propose to

• Expand pipeline template into full pipelines outside of GoCD server - that means in configuration repository plugin.
• Use existing, powerful templating engines to template the configuration elements. E.g. Ruby ERB, mustache, jtwig, anything user likes.
• Use templates for anything, not just full pipeline.

Some questions come along then..

If organization has templates to be shared among many projects (many repositories) then where should the templates be stored and hosted? For private pipelines and templates, this could be a repository with limited access. But there could be public templates too, users could be sharing pieces of configuration on github, just like we share libraries. (PS: @arvindsv I guess this is was what you mentioned once before).

How to reference reference remote configuration elements/templates from particular configuration repository? E.g. if you are a SCM-fanatic then you'll want to put templates themselves in a git repository. Then easiest way is to use a gitsubmodule to have your templates checked out in the configuration repo subdirectory, e.g. gocd-templates. Then configuration repository code looks like this:

# Super early draft!!
pipelines:
  my_dry_pipeline:
    group: rich
    label_template: "${mygit[:8]}"
    tracking_tool:
      $ref: "file://gocd-templates/issue_tracker.gocd.yaml"
    ...

But another method could be a http link reference like so:

# Super early draft!!
pipelines:
  my_dry_pipeline:
    group: rich
    label_template: "${mygit[:8]}"
    tracking_tool:
      $ref: "http://gocd-templates.example.org/issue_tracker.gocd.yaml"

The first method has gocd-templates as a git submodule (or could be a plain directory actually), if that is versioned with exact commit, then any update in module reference will trigger pipeline configuration to be reloaded in GoCD.
In second method if content served by URL changes, Go will not know about it, configuration will be stale until next commit is pushed. It might be undesired in some cases.

The right templating engine

I am currently looking into which engine would fit best here. It should rather have java implementation because GoCD plugins limitations. It should be powerful but also easy to use and learn.
Recently I discovered jtwig, I really like the modularity part of it.

If you have any suggestions, please say so.

Discussion

If you have managed 100+ pipelines then you know exactly all the shortcomings of GoCD pipeline templates. If you have some ideas on templating pipelines or some interesting use cases, then please share.

[johncmckim]

Will this include support for pipeline parameters? Templates and parameters really go hand in hand. Templates provide common build steps that can then be varied via parameters.

We are currently managing 50+ pipelines and are looking to bring in another 100+.

One of our use cases is a building -> testing -> packaging -> deploy .NET nuget packages. This process is almost exactly the same for all our .NET projects. The only variations are project names and folder structures. The template allows us to re-use the same build steps and vary the project names / folders.

Our other (more complex) use case relates to a more complex build process. We have a piece of software that is, unfortunately, built from two repositories, a common repo and a client specific repo. We use a pipeline per client that brings the two repo's together to build a client specific artefact. The templates allow us to have common build steps + parameters to vary the artefacts per client.

[arvindsv]

But there could be public templates too, users could be sharing pieces of configuration on github, just like we share libraries. (PS: @arvindsv I guess this is was what you mentioned once before).

Yes, it is close. I care about visualizing it, so people can say, "Yes, that looks like a configuration I want to use".

About your proposal: I think it's fine. You've summarized the idea well. You want to provide full file-level templating support. As @johncmckim said, there will be a need to pass in (in Ruby terms) the "binding", or at least some key-value pairs, to implement parameters, right? Otherwise, these templates will become very small pieces so that they include no params. With params, the templates can include more details and feel coherent. You need something like: (data or params) + template => pipeline.

One problem I can see, since this will not be using something standard like ERB is that it'll be up to the user to put together the template "in their head" so that they understand what the final pipeline looks like. So, you might need to provide a way to do that.

It will probably be time to consider non-Java plugin registration soon.

[tomzo]

You need something like: (data or params) + template => pipeline

Yes of course. My main point is that it can also be flexible like:

• (data or params) + template => set of interdependent pipeline. Something like @johncmckim 's second case, 2 pipelines which always appear together.
• (data or params) + template => just a job.
• (data or params) + template => 2 stages.

Parameters would be stored in configuration repository and become the ERB's Kernel.binding`, while templates would be stored in some shared repository and provide common templates to be expanded.

..put together the template "in their head"...

I think that even if it is ERB, then still a tool to preview what gets expanded with some example parameters should exist. This could emerge into testable templates too, e.g. assert that given these parameters applied on this template, expect configuration equal to (...).

It will probably be time to consider non-Java plugin registration soon.

@arvindsv Yes it would be helpful here, but until then, do you think a config repo plugin could leverage the fact that Go server runs on jruby and actually implement ERB templates within the server process?

[arvindsv]

My main point is that it can also be flexible like ...

Of course. Once there are templates, they can do anything. As long as the plugin understands what's happening and the final JSON or YAML or whatever is valid, then sure, one template can lead to multiple pipelines or just a job.

Yes it would be helpful here, but until then, do you think a config repo plugin could leverage the fact that Go server runs on jruby and actually implement ERB templates within the server process?

I don't think the JRuby classes are exposed to the plugin.

We were trying to once make it so that jruby.jar can be inside the plugin's lib/ directory, so that it can start its own instance of JRuby. But, that needs the OSGi part to expose sun.misc.unsafe etc. I don't think we continued too far down that path. It shouldn't be too hard to make it work, though.

I was thinking of having an HTTPS level endpoint for plugins to register, just like an agent would. Then, communication happens using JSON like usual. Maybe having a websocket open between the plugin and the server. Complications: Security (currently plugins can only start physically from the server box) and agent-side plugins.

I can take a quick look at trying to see what it takes to run a Clojure or JRuby-based plugin run in a few days, unless you or someone else has some time earlier than that.

[tomzo]

I was thinking of having an HTTPS level endpoint for plugins to register, just like an agent would

I like this approach. Then "plugin" can be really anything, just another independent application talking to go-server. Regarding security, the plugins could be using similar mechanism to agents, either a key or manual approval. Why do you think security is such a problem then? Are you concerned with intercepting the communication?

I can take a quick look at trying to see what it takes to run a Clojure or JRuby-based plugin run in a few days,

Yes please :) A proof of concept would be nice to get me started on this. I could pick it up from there and then write the plugin part to support some common scenarios with parameters.

[arvindsv]

Then "plugin" can be really anything, just another independent application talking to go-server

Exactly. It can be anything which can open a port, keep a connection alive (communication can happen from server to agent too) and respond using JSON.

Why do you think security is such a problem then? Are you concerned with intercepting the communication?

No. I'd just like the authentication to be a little more unique to a plugin, rather than just having one key (as agent auto register key is today). I was thinking of a per-plugin key. But, I'd need to think more deeply about the security it offers (does it?) vs. convenience. Either way, it's a decision we can take quickly.

Yes please :) A proof of concept would be nice to get me started on this.

Ok, I was thinking of a normal Java plugin, but written using JRuby or Clojure. Did you mean thinking about the non-Java plugin, or that? I think the JRuby or Clojure plugin can be easier and quicker for now.

[tomzo]

I was thinking of a per-plugin key. But, I'd need to think more deeply about the security it offers (does it?) vs. convenience.

That's was my first thought too. Personally I don't like that agents all share the same key, I think master key is an anti-pattern. I like how hashicorp's vault App ID solves this scenario. But that is a different topic.
For plugins I think a key per installation is most secure. Adding a plugin could be something like:

1. New plugin comes in for registration with a newly generated private key.
2. Go admin has to approve plugin. That causes public key to be added in go-server to trusted keys.
3. At any point in future plugin should be able to prove that it has the private key. Then it is plugin's administrators decision on how long to safely store it.

Did you mean thinking about the non-Java plugin, or that? I think the JRuby or Clojure plugin can be easier and quicker for now.

I was thinking to get jruby working in a java plugin. So that we can implement ERB templates. From user perspective one could write something like:

pipelines:
  my_dry_pipeline:
    group: <%= @group %>
    label_template: "${mygit[:8]}"
    tracking_tool:
      <%= @tool %>
  stages:
...
   <% if @include_some_stage %>
....

In longer term, with out-of-process plugins, it could be rewritten in ruby, or just run in its own process with jruby.

[arvindsv]

Ok. I have a flight to catch later and it has wifi. :) I'll try this.

[tomzo]

@arvindsv based on what @ketan says about loader jruby voodoo magic, that might be a reason to stop going in this direction. I mean it sounds that adding jruby to the plugin may make it unstable between jruby releases. I wouldn't want the plugin to be broken on half of deployments because of hacky setup needed for it to work. The main point was to get good file-level templates support, ERB was a candidate because it is well known and stable. We can move to other templating engine which has actual java implementation, if this is too hard.
In the long term out-of-process is what we should be working on, not a bunch of hacks to run ruby/clojure in server's process.
Nice to hear that you have some progress on clojure though.

[darioblanco]

Hi guys, I am handling my gocd XML config using Saltstack and Jinja templates (Python), I have a fully working example and it really suits this project, as Saltstack's pillar values are in YAML (equivalent to the config files for this plugin).

I know you haven't considered Jinja and there might be a reason for that, if not, I want to let you know that there is a Java Jinja implementation and I would love to see template and parameter support here (which is a must for me), then I would for sure switch to this plugin.

If you are interested, I can share my Jinja template with you, which supports templates and parameters. Of course, there would be some work to match the YAML specification for this project, but I don't think is too much.

[arvindsv]

Nice. @tomzo is away for a little bit, I think. He'll probably reply once he's back.

[tomzo]

@darioblanco thanks for suggestion. Please share your template and some examples. If you have been already managing pipelines using file-level templates as you say then we could use some of that to craft a few test cases.
Do you have any user experience with Jinja java implementation you have pointed? There are some issues in that repository, I don't know if these cause significant limitations.
I don't know Jinja, nor Jtwig which I have suggested so far, but it seems to me they provide similar features. However Jinja is was originally created for python, then ported to java, while Jtwig was created for java.
I like your idea in general, but do you have any specific reason to use Jinja rather than other templating language?
Lastly, please clarify to me - If you move to using this plugin then would you still need Saltstack to manage pipeline configurations? I guess not.

[darioblanco]

Hi @tomzo

I have experience with Jinja from my Flask/Django days, but not with its Java implementation. I would personally use a native Java solution if is available, I just don't know what are the limitations for adopting templates and parameters into this project.

My reason to use Jinja is the following: as I am provisioning my gocd instance with Saltstack, I wanted to have the configuration as code, and I dislike XML a lot. As I already use Jinja for Saltstack templating, I thought it was a good idea to just create a cruise-config.xml file as a template and support as much as possible from GoCD configuration reference. I already have experience with Jinja, so it was relatively easy to create a template.

My Jinja templates currently support:

• Cruise parameters: xmlns_xsi, xsi_no_namespace_schema_location and schema_version
• Users and roles (based on a passwords.properties file)
• Repositories
• Pipelines
• Templates
• Environments

I will share the config in another post (it is divided in different template files).

If this plugin supports templates and parameters/environment variables I will simplify my Saltstack configuration a lot, as I will move the pipeline/template definition to a different repository and saltstack will just have to add the repository load tag into cruise-config.xml. So yes, Saltstack won't need to manage pipeline configurations :)

[darioblanco]

About my example, I have created a gist: https://gist.github.com/darioblanco/f5d9840d91a875d92ac9b718ee128f6d

The YAML file sadly doesn't match your reference. It can be adapted with strong Jinja changes though.

For this project I guess the only interesting files are templates.jinja and pipelines.jinja. I decided to share the other files as well to give some context.

I supported as much as possible for my specific use case, not all the attributes from the GoCD configuration reference, I just add them in my template on demand, so imagine how helpful would be to use your plugin :)

[moritz]

Hi all,

just another user's perspective:

• Having some form of templating is quite crucial. We have pipelines for ~50 pieces of software, and maintaining the pipelines individually is not feasible.
• I see two main weaknesses with GoCD's built in XML templates for pipelines:
  • you cannot template everything. For example materials cannot be part of templates, and parameters aren't supported in resource specification of jobs
  • Lack of conditionals. Some of our pipelines have an extra stage for additional tests, and currently we have to maintain a separate template for those that do and those that don't (or introduce dummy test runs with /bin/true, which adds bloat).

Any templating approach that improves on those points would be a big plus.

As a user, I've had experience with Jinja2 (mostly coming from Ansible), and it's pleasant enough to use. However it's mostly meant for HTML or text file output, whereas for the pipeline configuration, the result really is a data structure, encoded in YAML. I wonder if there is something better that works on the data structure level, not on the text level that is then again parsed as YAML.

Thanks for all of your work on this,
Moritz

[vaibhavparnalia]

+1
We have an app release workflow with ~25 pipelines which are based on 4 templates. Now we want to have same workflow for ~20 apps. If we are not able to refer these pipelines to a template as pointer, any new introduction in the workflow will be challenge.

[davidkennedydev]

I'm doing templates using Mustache. If it's usefull to someone can I publish examples.

[fire]

@davidUser Can you post examples?

[tpidor]

I was searching for this template feature some 6months ago when we started switching to Pipeline As Code in GoCD. Having looked at this issue #2, we decided to go with j2 templates and wrote a little python script to generate the yaml pipelines (1 gocd.yml = 1 pipeline) based on j2 template and variables. Since then, I managed to create 19 different j2 templates for the 71 pipelines and counting. We thought this is much better than the GoCD template which only looks after of the stages. Whereas the implementation we did is to j2 template the entire pipeline including materials, upstream/downstream dependencies, etc. I might get trouble with the company if I post examples but I would suggest looking at python jinja2 lib.

[stCarolas]

from gitter

we made pipelines which consume git repo url as parameter and use jinja2 for generating base *.gocd.yaml
in these yaml files we use shared bash scripts as additional material
like that

format_version: 2
pipelines:
  PUBLISH_corp-cards-ui:
    group: CORP-CARDS-UI
    label_template: ${code[:10]}
    materials:
      code:
        git: http://git/scm/corp-cards/corp-cards-ui
        destination: code
        auto_update: false
      gocd-scripts:
        git: http://git/gocd/gocd-scripts-nodejs
        destination: gocd-scripts
        blacklist:
        - '**/*.*'
        auto_update: false
    stages:
    - test:
        elastic_profile_id: nodejs
        tasks:
        - script: 'export CODE_PATH=$(pwd)/code; 
                       export GOCD_SCRIPTS=$(pwd)/gocd-scripts;
                       $GOCD_SCRIPTS/front_test.sh'
    - publish_docker_image:
        elastic_profile_id: nodejs
        tasks:
        - script: 'export CODE_PATH=$(pwd)/code;
                   export GOCD_SCRIPTS=$(pwd)/gocd-scripts;
                   $GOCD_SCRIPTS/docker_build.sh '

such way with using shared scripts in stages without templates allows customizing only one stage from casual flow and getting updates for other stages
also if one pipeline breaks with new version of scripts we can run it with old script version by Run With Parameters

[davidkennedydev]

@fire Here is the example, sorry by the while but i need remove a lot of restrict information from the example.

https://github.com/DavidUser/template-gocd-mustache-example

[moritz]

I've done an experiment to use the jsonnet data template language for the JSON plugin, and written up a summary in the wiki.

[fire]

Looks useful, I'll try it with my pipelines.

[thatsk]

@moritz looks like the example is not in link. & don't referred to config yaml pluing can you more elaborate like @davidUser

[thatsk]

@davidUser i like your approach. But there are two problems now.

1. If we are going to use this example. And we created 100 of service pipeline to use generic template.
   how do we refer to particular component service. if we are referring this mustache template. because mustache templates are in different repo. and service don't have pipeline template configuration

[thatsk]

@davidUser config repo plugin will look for only changes. and once we did with change for service component repo. how do we automatically trigger for template code with some manual paramter?

[fire]

I did something similar with jsonnet. For the gocd to scan for changes, all the relevant repositories must be added as watched materials.

[Edited] If the output json is different that will cause a reload.

[thatsk]

is there any example?

[thatsk]

let say component.repo has ci.gocd.yaml and i want to make generic templates for the same component which is of 200. Config repository plugin looks for ci.gocd.yaml.
but if i generated template for it. and moved to new repo which has only pipeline code.
and if i make changes to component how can i get the code from generic template repo code.
and make things work.

[davidkennedydev]

@thatsk I think that a better approach is maybe use some programmatic interface and write your business rules to pipeline generation. My example is just a simple workaround that I use to manage simple bootstrap to pipeline configurations.

[arvindsv]

There's some related work being done by @marques-work here: gocd/gocd#6123 (comment)

[timothy-cloudopsguy]

Forked and added some code showing what it could look like to have true YAML templates. This allows you to use existing old/archaic template: from GoCD, but also use a new template_from_repo: that gives extreme portability while honoring the minimalistic styling of YAML that tomzo gives us. No need to know how to convert a YAML minimized template into the robust JSON that GoCD API needs in order to actually upload it and honor it. And, these templates live where you want, instead of being thrown into the cruise-config.xml file.

The template only contains stages and can fully utilize any/all environment variables that are in the main portion of the caller YAML file.

This is a very crude, but fully working, demonstration of what is possible. We are using it at my company right now with success. Best of luck.

https://github.com/timothy-cloudopsguy/gocd-yaml-config-plugin

At a high-level, we just tapped into the PipelineTransform.java file, at the point where it finds the template: marker in the YAML file, we now look for template_from_repo: and add anything in that file (i.e. all stages) to the steam. It's basically just concatenating the files together at this point, but I think it's pretty slick.

        addMaterials(pipeline, pipeMap, formatVersion);
        //
        // If pipeMap contains YAML_PIPELINE_TEMPLATE_FROM_REPO.. grab the stages from the template
        // file found in the repo
        if (pipeMap.get(YAML_PIPELINE_TEMPLATE_FROM_REPO_FIELD) != null) {
            Object templateRepo = pipeMap.get(YAML_PIPELINE_TEMPLATE_FROM_REPO_FIELD);
            if (!(templateRepo instanceof String))
                throw new YamlConfigException("expected a string value in template_from_repo");
            String repo_file = (String) templateRepo;
            LOGGER.info("transform(): Attempting to load stages using template {} from repo {}", repo_file, gitHelper.getRepoUrl());

            Map<String, Object> templatePipeMap = getTemplateFileFromRepo(gitHelper.getWorkingDirAbsolutPath(), gitHelper.getBasePath(), repo_file);            
            addStages(pipeline, templatePipeMap);
        }
        else if (!pipeline.has(JSON_PIPELINE_TEMPLATE_FIELD)) {
            addStages(pipeline, pipeMap);
        }

[wilsonr990]

@fire Here is the example, sorry by the while but i need remove a lot of restrict information from the example.

https://github.com/DavidUser/template-gocd-mustache-example

For whoever is of interest, similar to @davidUser's approach I have implement in my project templating using yaml aliases. Side effects:

• All files need to be joint for gocd plugin to be able to parse the aliases. I solved that with a script that I trigger whenever any of the files is edited.
• These templates are not gocd templates. I'm rather solving the problem of being able to reuse and prevent duplications

See links below:

• Folders with pipelines and templates
• Simple script to generate file parsed by gocd plugin

I hope it's useful!!