(Feature Request) Manually trigger deployment

[Marcel2508]

What do you think about a feature to manually trigger deployment for successful builds.
Where you can decide the desired deploy-target (dev, staging or prod systems for example).

Similar to how gitlab works.
Automatic deployments shall still be possible.

[jkuri]

This seems like a very nice feature to be hones, can you please post some screenshots from gitlab how it is achieved there and I will see what I can do.

[inkvizitor68sl]

CC: @jkuri @Porsh33

I have no configured Gitlab CI right now (btw, thanks again for freedom from gitlab =)), but made some screenshot, how it can be implemented in Abstruse CI with minimal amount of code added:

Basic idea that "paused at start" (see gray label at screenshot) jobs need to be implemented to "matrix:" structure.
So Abstruse will automatically "build" something, then real human will use "restart" button on deploy job (ofc, it can be new "Play" button, if it will be added).

So, we will be able to create matrix like below and manually trigger pipeline steps after 'build' will be completed.

  -
    env: SCRIPT='build'
    image: ubuntu
  -
    env: SCRIPT='deploy_to_test_and_run_test' 
    image: k8s_admin_image
    start: manual
  -
    env: SCRIPT='deploy_to_prod'
    image: k8s_admin_image
    start: manual

After that "quick" realization of basic CD, some "big-CD" features can be added:

• option to block "Play" button for next steps until previous will be completed (dependency system for jobs inside build)
• option to trigger next Job automatically when previous job completed - to be able auto-deploy to test, but manually deploy to prod after test. It can be done via YML, but can be more comfortable to split build and test deploy to separate jobs.

Sorry for strange english, i talked with @Porsh33 in offline on native lang, he understood idea, probably he will make later some pull request with basic realization.

[jkuri]

I generally like the idea, but what would be the build status until you run those manual jobs? We can also make that deployment jobs to be triggered after all test jobs are passing and also you can define specific branch from where to deploy in .abstruse.yml configuration.

Do you have something in mind how we can define configuration for that in .abstruse.yml?

/cc @Porsh33

[inkvizitor68sl]

but what would be the build status until you run those manual jobs?

I think, that build status should represent status of really executed jobs. Small mark on "passing" status (or near it) can show, that pending jobs exists, for example:

• "passing" green status - all jobs executed
• "passing*" or "passing >" green - manual jobs still exists (maybe some "less green" color).
• "running" - any job running, not matter manual or auto-started
• "failed" - any job failed (manual or auto)

It can be good behaviour, until (and if) Abstruse will be focused much on continuous delivery.

Really difficult question here - what other users will expect from after_failure/after_success hooks. I cant imagine good behaviour for this (probably most expecting - to run this hooks in each Job#, so users will use "if-then" to run exact fail/success script after job).

Do you have something in mind how we can define configuration for that in .abstruse.yml?

Yes, i will try to describe as yaml example below.
Small notice - some of this is not easy to code (and anyway it is a lot of new code), but i looked through current sources and hope now that it all can be done without refactoring or any breaking changes.
It is only vision "how CD can be done in Abstruse CI without breaking changes" from my side, not feature request or something like. I am trying to play around current matrix: feature.

image: docker.io/ubuntu/ubuntu

matrix:
- name: build_xenial #some uniq name of job
  image: docker.io/ubuntu/ubuntu:xenial
  env: SCRIPT='build_and_unit_tests'
  start: auto #default value
- name: build_focal
  image: docker.io/ubuntu/ubuntu:focal
  env: SCRIPT='build_and_unit_tests'
  start: auto #default value
- name: deploy_test
  image: k8s_deployer_image
  env: SCRIPT='deploy_to_test'
  start: auto
  depends_on: [ build_focal, build_xenial ]
  only_trigger_on: [ ".*", PR, tag ]
  # only_trigger_on overrides settings from web-interface
  # normally it is webhook settings, but here we limit webhook events for exact matrix member here
  # have to be documented carefully - we cant ignore PR on settings page, if want to use it here (webhook will not be triggered)
  # "quoted" - is names of branches
  # PR/tags - keywords for pull requests and tags
- name: deploy_prod
  image: k8s_deployer_image
  env: SCRIPT='deploy_to_prod'
  start: manual
  depends_on: [ deploy_test ]
  only_trigger_on: [ "master" ]

# this part used to skip any jobs
branches:
  test:
    - ...
  ignore:
    - ... 

script: 
  - if [ "${SCRIPT}" = 'build_and_unit_tests' ]; then /build/build_and_unit_tests.sh; fi
  - if [ "${SCRIPT}" = 'deploy_to_test' ]; then /build/deploy_to_test.sh; fi
  - if [ "${SCRIPT}" = 'deploy_to_prod' ]; then /build/deploy_to_prod.sh; fi

after_failure:
  - if [ "${SCRIPT}" = 'build_and_unit_tests' ]; then mail "Build failed"; fi
  - if [ "${SCRIPT}" = 'deploy_to_test' ]; then mail "Deploy to test failed"; fi
  - if [ "${SCRIPT}" = 'deploy_to_prod' ]; then mail "Deploy to prod failed"; fi

So how it should be executed:

• 2 jobs started in parallel (build_xenial/build_focal) right after build created
• deploy_test job created in "Blocked" state (not passing it yet to workers API)
• deploy_prod job in "Pending + Blocked" state added if it was commit to "master" branch. Blocked state has more priority in web UI.
• when 2 jobs (build_xenial/build_focal) completed - job "deploy_test" moves to worker queue
• when deploy_test completed, "Blocked" flag deleted from deploy_prod job, now it has active "Play" button and "Pending" state at UI.
  (separate name needed for "Pending because manual job" state, because actually "Pending" is about "no free worker yet for this job", but i didnt find good one).

About new variables in matrix:

• name - is name of job, needed to implement dependencies between jobs. For backward compability can be missing (such matrix members just cant be used as dependency for other jobs).
• start - manual or auto (auto, if not set). If auto - such job will be added to workers API/queue, when all "depends_on" jobs done. If manual - we need to hit "play/restart" button to start this job.
• depends_on - job names from matrix, which have to be done (successfully) to start current job. Until dependencies completed - "Blocked" status can be used to show that in interface
• only_trigger_on - list of commit events (commit/PR/tag), when exact job (matrix member) have to be added to build. Important thing, that we cant "add" here some trigger - mean that if event ignored by webhook settings or in "branch:" section, then it will be still ignored, even if listed in only_trigger_on
  (all names are symbolic ofc)

Also to do usage of that more comfortable some global variables need to be added:

• build number - to use it as uniq version tail, when git hash cant be used. Job# can be received from docker hostname, but it is not same for jobs in matrix. (cant build v1.${JOB_ID} and deploy it in another job, but can do this with v1.${BUILD_ID})
• name of job (if set in matrix) - usefull for scripting outside .abstruse.yml (however it can be duplicated at matrix env)

/ cc @jkuri @Porsh33