ubuntu-noble monit replacement proposal

[jpalermo]

Things that are NOT part of the plan

• Removing monit
• Requiring changes from release authors to run in a noble stemcell

Proposed Plan

• Bosh release jobs optionally have a top level file called processes.yml
  • This file has an identical schema to bpm.yml
  • This file can be a relative symlink to the existing bpm.yml template
  • This file can be templated via normal bosh templating
  • The old bpm.yml template is no longer required, but can be kept for backward compatibility with older stemcell lines
• The BPM binary will be included on the stemcell
• The agent will use the contents of processes.yml to construct a systemd service for each process listed in the file
  • The service will essentially /var/vcap/bosh/bin/bpm start JOB_NAME -p PROCESS_NAME
  • The service will be configured to watch the PID contained in the file dropped by bpm
• If the release does not contain a processes.yml file, monit will be used

Potential Future Improvements

• Additional escape hatches for processes
  • Some jobs have not migrated to BPM because of restrictions imposed by it. Those jobs can continue to use monit for now and we can add additional BPM functionality to support their use cases
• Additional monitoring options in processes.yml that are used to configure systemd for more advanced non-pid monitoring
• Additional bosh agent commands that can be used to gather state for configured processes
  • This is meant as an alternative to monit summary that would show you details for both monit and systemd managed processes

Places that would need to be updated

• bosh-agent
  • A systemdJobSupervisor will need to be created to live alongside monitJobSupervisor
    • Both will need to be active at the same time so work will need to be done to support that
• bosh-cli
  • Rendering templates for create-env would need to also render the processes.yml file
  • Generating a new job via bosh generate-job would need to seed the processes.yml file
  • The release/job/dir_reader.go would need to account for including the file as part of a job
• bosh-director
  • validate_monit in src/bosh-director/lib/bosh/director/jobs/release/release_job.rb will need to support either file existing (but probably at least one of them)
  • Several spots in src/bosh-director-core/lib/bosh/director/core/templates/ where the templates are found and rendered

[jpalermo]

Related Issue

[aramprice]

Some jobs have not migrated to BPM because of restrictions imposed by it.

Is there a list of these issues anywhere? BPM's unsafe was intended to accommodate these sorts of cases.

[jpalermo]

Nothing concrete as far as I know. Hopefully a long transition period will give the flexibility for any issues to be discovered before monit actually goes away.

[rkoster]

@poblin-orange do you have any more context on why the k3s bosh release moved away from bpm? orange-cloudfoundry/k3s-boshrelease@ed211c2

[rkoster]

The kubo-release, also seems to not be using bpm: https://github.com/cloudfoundry-incubator/kubo-release/blob/master/jobs/kubelet/monit This is still true for the private fork: https://github.com/pivotal-cf/pks-kubernetes-release/blob/main/jobs/kubelet/monit (file is unchanged, but only accessible to Broadcom employees).

@YanzhaoLi do you happen to have more context on why kubelet is not being run via bpm?

[YanzhaoLi]

We have a lot of jobs not being run via bpm. I'll try but I am not sure if I can find the initial context.