Pre/Post Build Jobs

[jonaswre]

We sometimes have the requirements to run additional steps before/after the Business Central compile starts.

A usecase for this is for example to compile a markdown documentation or package other build artifacts.

My current implementation uses NewBcContainer.ps1 to execute code before the compilations starts. But this is pretty limited.
You are stuck in windows with powershell.

You won't be able properly integrate with github. For example its not possible to publish additional artifacts.

I've created a prototyp that should check if a PreBuild.yaml / PostBuild.yaml exsits in the .github/workflow folder.
But this has a pretty bad limitation. Every local file you reference needs to exsits if you want to use it in uses:.
actions/runner#895

Even including an if in the workflow doesn't help.

So to workaround this limitation, you will need to use the same mechnism thats used for the schedules.

You need to create a new property in the AL-Go-Settings.json when you run the UpdateSystemFiles Workflow you will need to create the workflow trigger in CICD,Current,NextMajor,NextMinor.

{
  "type": "AppSource App",
  "templateUrl": "https://github.com/microsoft/AL-Go-AppSource@preview",
  "nextMajorSchedule": "0 0 1 * *",
  "nextMinorSchedule": "0 0 1,15 * *",
  "currentSchedule": "0 0 1,15 * *",
  "preBuildWorkflow: "./.AL-Go/PreBuild.yml",
  "postBuildWorkflow: "./.AL-Go/PostBuild.yml"
}

One thing im not sure yet. I only need to run something before and after the Build Matrix. But when this is implemented. Some People might prefer to have this run before every Run Pipeline.

I think in general is should be pretty easy. But before I start, I wanted to get some feedback on this.

[jonaswre]

@freddydk we talked about this at the tech days.
You think this is the right direction to go in?

[freddydk]

Thanks for the idea.
Yes, I remember our discussion - will put some thought into this and get back with my comments tomorrow.

[jonaswre]

This would also allow you integrate different technologies like the PowerApps integration.
You could just have a pre/post workflow for that.

[freddydk]

I like the idea of having places where you can inject custom jobs or steps, but it needs to be safe, both from a security perspective and from an upgrade perspective - not being broken every second day.
I think I will need to build a small prototype to see whether this idea holds.

[freddydk]

Maybe use PowerPlatform thingy as vehicle for the prototype - might be a good idea.

[jonaswre]

I think safety from a stability point can be archived by anchoring it to the "Build" Job. That way the only thing that needs to stay is the id of the Build Job.

$yaml.AddBefore('jobs:/Build:', ["name: PreBuild", "uses:  '$($settings.preBuildWorkflow)'"])

$yaml.AddAfter('jobs:/Build:', ["name: PostBuild", "uses:  '$($settings.postBuildWorkflow)'"])

Regarding security; This is an advanced usecase, simular to the ps1 overrides. AL-Go should be secure by default. But if an advanced user runs malicious code in an powershell or as a github action its on them. But it obviously need to be ensured that these files can't be changed in a public repo with a pr.

But that should be the same mechnisms used to secure the workflow itself.

This could also be the first usecase for a ".AL-Go" Repo ;-) just as a container for reusable pre/post workflow.

And as for step vs job. I think it would be more useful to be able to add jobs. Because this would allow you to paralize if needed or choose a other runner.

If you need help with the prototype I could carve out a few hours.

[jonaswre]

One more thing that will be nessary, at least for the prebuild.
The Build Task needs to download the artifacts uploaded from the preBuild Job.

For my prototype i've done this by defining a static artifact name. (prebuild-artifacts) The build task then downloads the prebuild-artifact to the workspace root. This allows you to copy files to multiple locations in workspace.

Usecase for this would be the translations, we talked about aswell.

To handle translations you could create a prebuild workflow. That does a "fast compile" to generate the .g.xlf then translates that and writes the files back to all apps. The regular build would then automaticly use the by the prebuild translated files.

[freddydk]

The Build job is a matrix and there can be several of those - I lean more towards adding steps in the reusable workflow (_BuildALProject) which is used from all build workflows (CI/CD, PR, Current, NextMajor and NextMinor)

The problem with jobs is that they run on their own agent - meaning you cannot download files like dependencies in a job. We would have to support both jobs and steps I think.

[jonaswre]

Because of the Matrix and my use case worked well with pre and post. I opted to Run them before and after the build.

I think that's already pretty powerful.

There are probably different requirements for different use cases.

1. Before/after all builds (a job)
2. Before/after every Project is build (steps)
3. Maybe even before after every App ist build. (steps)

For compiling/packaging software that is only loosely connected like for example power apps you would use option 1.

To aquire app/non app dependencies like xlfs, extraction of build artifacs or custom checks you would use 2./3.

The problem with jobs is that they run on their own agent - meaning you cannot download files like dependencies in a job. We would have to support both jobs and steps I think.

I don't really understand the issue. I've downloaded GitHub packages in the PreBuild job running on ubuntu-latest. The token you have in the workflow has just the normal permissions.

Or do you mean, since you are running in a different agent you can't reuse the dependencies and they would need to be redownload. If so that's the reason I used download dependencies with path: '.'

That should merge the output of the prebuild with the build. Might not be the most efficient but it gives you all the flexibility you need.

[freddydk]

I did create a small prototype. Not sure this will get implemented or not - but if you create a test repository (do not use your prod repos for this test) and Update AL-Go System Files with freddydk/AL-Go@placeholders as your templateUrl - then your _BuildALGoProject.yaml file will contain some placeholders.
Whatever steps you insert between these placeholders are preserved when updating AL-Go System files later, like:

      - name: PreBuild.Start
        if: false
        run: |
          # All steps between PreBuild.Start and PreBuild.End will be preserved when updating AL-Go System Files
          # Note that changes to the base workflow might cause your steps in this section to break, please update them accordingly

      - name: Custom step
        run: |
          Write-Host "This is my custom step"

      - name: PreBuild.End
        if: false
        run: |
          # End of custom PreBuild block

The Custom step will survive Update AL-Go System files because it is between the PreBuild.Start and PreBuild.End.

Note that it is absolutely NOT certain that this will be implemented - it is just a test for now.

The biggest problem I see with this is that people will have to make the same modifications to all their repositories - many partners would want to have the same rules for many repositories.

The biggest advantage I see with this is that it is easy to develop and test your custom steps - you are allowed to modify your workflows in pre-defined areas.

[freddydk]

The current prototype is not a job - it is a series of steps in the _BuildALGoProject reusable workflow.

I was going to see whether the same mechanism could be used for jobs, but it is a but more complex (due to dependencies between jobs)

Could you explain to me why it needs to be a separate job and what it does?

BTW - It is possible to have a fork of the AL-Go-PTE repo - add your customizations to that - and use that as your template repo, that works poorly with the current prototype really - the customizations in that template repo should NOT be inside the Start -> End placeholders.

[jonaswre]

I was going to see whether the same mechanism could be used for jobs, but it is a but more complex (due to dependencies between jobs)

Because of that issue I limited it to only one job before and one after build.

Build is dependent on PreBuild and Post build is dependent on Build. (If any of them exists.) And the mechanism I used to transfer files from PreBuild to Build is simply by uploading artifacts to a predefined name and them downloading that in Build.

Could you explain to me why it needs to be a separate job and what it does

We are designing complex control addins to improve UI/UX.
We have developed some custom tooling and it's significantly faster on ubuntu then on windows, linux is cheaper and matrix might come in handy sooner or later. So thats why I need to build in a different runner and that result needs to be compiled before the AL compiler starts.

And I have some more use cases in mind that might require a similar pattern.

Adding steps makes it easier to debug if something goes wrong then using PowerShell scripts but they still limit you to the capabilities of your runner.

BTW - It is possible to have a fork of the AL-Go-PTE repo - add your customizations to that - and use that as your template repo...

That has a few disadvantages. Depending on where you have your customization you will encounter merge conflicts. (As you said you need to make sure they don't break every second day). And the fork approach will probably not work for files that live outside of folders that are already "maintained" by AL-Go . So you will likely not be able to manage for example common workspace settings, script overrides, app.json or similar.

Specially in multi project repos you need custom tooling for that.

[freddydk]

Yeah, just tested out - new jobs in a workflow can be added anywhere - they don't need to be in any specific order.
I will add new functionality to the prototype allowing you to add jobs to any workflow - and maintaining the needs.
i.e. if you modify CI/CD and add a job (which you add to the needs of build - then that needs should be maintained).
will try:-)

[freddydk]

I added the ability to add jobs to the prototype.
If you add a job called CustomJob: to one of the YAML files - that will also be preserved when doing Update AL-Go System Files.... - or rather, it will be moved to the end of the workflow - but the needs dependencies will be maintained.

Try it out:-)

[freddydk]

This repo: https://github.com/BusinessCentralApps/buildorder contains 2 extra jobs in the CI/CD workflow.

[freddydk]

This PR now contains an implementation suggestion of this: #715

Have a look at the description and try it out.

https://github.com/freddydk/AL-Go@custom is the template URL you can try out.
You can also use my indirect template repository: https://github.com/freddydk/MyIndirectPteTemplate, which contains some customizations and then you can add your own in the final repo.

This is the way I envision that you can add your customizations and scripts to a indirect template repo and use that as the repo.