Only send a limited number of jobs at a time to the cluster to avoid stressing it too much

[kit-ty-kate]

@talex5 would this help?

[talex5]

Yes, but we do want it to use the cluster fully when the cluster isn't busy. Having 500 jobs at once is fine, as long as nothing else needs to use the cluster at the same time.

Another option might be to mark all revdeps jobs as non-urgent. But really, we need more priorities, e.g.

1. ocaml-ci or opam-repo-ci main job (new commit pushed)
2. opam-repo-ci revdeps
3. ocaml-ci or opam-repo-ci update (rebuild due to an opam-repository merge, etc)
4. health-check build
5. base builder image update

[kit-ty-kate]

Agreed. I opened #25 in the meantime to set all revdeps jobs as not urgent.

[talex5]

Now that ocurrent/ocluster#88 this PR shouldn't be needed.

[kit-ty-kate]

It looks like this proposal may be necessary to avoid taking more RAM than the host server can handle (e.g. when sending > 30_000 jobs at once)

[talex5]

Worth a try. Though I'm not sure where all the memory is actually going. opam-repo-ci has memtrace support compiled in, so it might be an idea to turn that on and find out.

[kit-ty-kate]

for some reason this makes opam-repo-ci stop doing any work. /jobs shows all the jobs as (ready to start) but none are actually starting.
Something to do with the way ocurrent is handling caches maybe?

EDIT: nevermind, it was: ocurrent/ocurrent-deployer#92