Add HTTP health-check server

[faraquet]

The tiny HTTP health-check server that now runs as a supervised process.

• Endpoints:
  • / and /health:
    • Returns 200 OK with body OK when the supervisor and all supervised processes (workers, dispatchers, scheduler, and the health server itself) have fresh heartbeats.
    • Returns 503 Service Unavailable with body Unhealthy if any supervised process (or the supervisor) has a stale heartbeat.
  • Any other path: returns 404 Not Found
  • Configure via config/queue.yml under health_server:. Both host and port are required.

Enable and configure via process configuration:

production:
  health_server:
    host: 0.0.0.0
    port: 9393

Note:

• This runs under the supervisor just like workers/dispatchers.
• When the Puma plugin is active (plugin :solid_queue in puma.rb), the configured health server is skipped to avoid running multiple HTTP servers in the same process tree. A warning is logged. If you need the health server, run Solid Queue outside Puma (for example, via bin/jobs) or disable the plugin on that instance.

[faraquet]

@rosa could you please have a look? This is a simple and safe change, but I believe it will be useful for many people, including myself.
I'm sure the failing tests are unrelated, as I've seen the same issue in other branches.

Happy to discuss if needed

[Th3-M4jor]

Feel free to disagree with me on this, but what about adding some kind of check to ensure that this can't be run while also using the Puma plugin?

[faraquet]

Thanks for the hint, @Th3-M4jor 👍

I agree with you and will add an extra check.

[faraquet]

Added ✅

[faraquet]

I've revised the idea in favour of launching via supervisor and setting it up via a config file, just like workers

[rosa]

Hey @faraquet, thanks for working on this! However, I'm not sure the separate web server process guarantees anything about the health of the workers and dispatchers and the other processes beyond the supervisor being alive, which you can check via the pid 🤔

Another question I have is why the possibility of running multiple health server processes per supervisor.

[faraquet]

Thanks a lot for the feedback, @rosa!

Another question I have is why the possibility of running multiple health server processes per supervisor.

I was thinking about doing something similar with the workers and reusing their configuration, but I completely agree - there's no reason to run multiple processes, so I changed it to use just one.

I'm not sure the separate web server process guarantees anything about the health of the workers and dispatchers and the other processes beyond the supervisor being alive, which you can check via the pid

In our container setup, it's quite inconvenient to check via pid, and having an HTTP response would make things much easier.

What do you think about making the server a bit more advanced and actually checking the supervisor's state? I've made the POC version already. If you think it looks promising, I can continue developing it and add more tests.