Add health check API endpoints

[oneonestar]

Description

Add two new health check API endpoints: /trino-gateway/livez and /trino-gateway/readyz.

Additional context and related issues

/trino-gateway/livez always returns 200.
/trino-gateway/readyz returns 200 after the initial fetch from database and the first round of Trino cluster health check is completed. Otherwise, 503 will be returned.

Release notes

(x) Release notes are required, with the following suggested text:

* Added API health endpoints `/trino-gateway/livez` and `/trino-gateway/readyz` for monitoring liveness and readiness. ({issue}`issuenumber`)

[oneonestar]

Might need to bind ActiveClusterMonitor by default instead of injecting it in config.

[mosabua]

Have not really looked at the PR but I think we should agree on same names for those probes in Trino Gateway and Trino . What do you think are suitable URLs .. I think it could literally just be ready and live, or liveness and readiness .. or is there some standard that anyone is aware of?

@nineinchnick @martint @dain @electrum ?

[nineinchnick]

AFAIK Trino doesn't yet have endpoints dedicated for probes. The z suffix is a convention started at Google, widely adopted in Kubernetes apps. I'm ok with that, OTOH there's little chance we'll have conflicts with other endpoints.

[mosabua]

AFAIK Trino doesn't yet have endpoints dedicated for probes. The z suffix is a convention started at Google, widely adopted in Kubernetes apps. I'm ok with that, OTOH there's little chance we'll have conflicts with other endpoints.

Fair enough .. i saw the healthz in others but the two you created @oneonestar are better .. so lets run with that then.

[mosabua]

Suggested change

	`/trino-gateway/livez` returns status code 200, indicating the server is alive.
	* `/trino-gateway/livez` returns status code 200, indicating the server is alive.

we should detail what "alive" means specifically for Trino Gateway ..

[mosabua]

Also .. can it return something else?

[oneonestar]

This endpoint always sends back a 200 status code. However, it might not respond if the gateway is too busy, stuck, or taking a long time for garbage collection. This is what "alive" means.

[mosabua]

I know this is k8s basics but I think we should let our users know anyway .. not all of them will have k8s background or even run it on k8s

[mosabua]

Suggested change

	`/trino-gateway/readyz` returns status code 200, indicating the server has
	* `/trino-gateway/readyz` returns status code 200, indicating the server has

also need to detail what that means .. such as .. the backend database connection is working, the connection to the external routing server (if any) is alive, and so on

[oneonestar]

In current implementation readyz means the initial connection to backend database and the first round of health check on Trino clusters were completed. I think this might expose too much detail to the user.

[mosabua]

No .. I think our users are very technical and we should not hide that information from in the docs .. its not in the UI or so .. but it needs to be in the docs