Pod runtime information is in a non-sensical spot

[mook-as]

Currently, in the role manifest:

• the manifest has multiple instance groups
• each instance group has a bunch of jobs
• each job has properties
• inside the properties for each job, there is a bosh_containerization section
• inside of that, there is the run information, such as limits, affinities, and scaling information

This makes no sense: you can't scale each job apart from the pods ( = instance groups) they are in. They should be moved to under instance groups (though we may want to keep limits per-job in case we get to container-per-job). Otherwise it's unclear what happens if I have two jobs in an instance group, one saying I can't HA it, and one saying I can…

[jandubois]

We could still compute the intersection of the requirements of all the jobs and use that to configure the instance_group. We already do this for the PSP.

And if the intersection becomes empty (one job must be a singleton; the other requires at least 2 instances), then validation can throw an error.

Instance groups are really just an artifact of the virtual machine view of the world, so I would prefer to deemphasize them as much as possible. I would hope that eventually we would replace them with affinity rules and have just one job per instance group. Then each job becomes a pod, and the different processes inside a job become separate containers with a shared filesystem.

[viovanov]

The runtime information is in a nonsensical place because:

• we define runtime information to be per instance group (which is a bit wrong, as @jandubois mentions - an artifact of the VM world)
• we must be compatible with BOSH manifests - so that's why we can't move them to be per instance group @mook-as

IIRC right now, if you specify runtime information in in more than one job, you'll get an error.
So we require you to have one runtime block per instance group.
In the future, we want to describe jobs more than we describe instance groups, and we want to use more BPM information (if possible).

With the current BOSH manifest schema [1], I don't think there's a nicer way to achieve what we're trying to do, given that per instance group properties are deprecated.

[1] https://bosh.io/docs/manifest-v2/#instance-groups