How can I get health of rr server - grpc plugin?

[lanphan]

Hello,

I'm using RR to provide grpc services, and it works well, thanks so much.
However, lately I get 1 issue: load balancer in k8s sends requests to rr instance which doesn't have any free worker, it leads to error "No free worker" response. I think this can be prevent if we have healthcheck for grpc service like http service I see somewhere on Internet. Would you please tell me how I set it up?

Current my setting:

• rr version: rr version 2.6.4 (build time: 2021-12-07T13:25:47+0000, go1.17.4)
• Linux ubuntu container
• my rr.yml

rpc:
listen: tcp://127.0.0.1:6001

status:
address: 127.0.0.1:2114

server:
command: "/usr/local/bin/php /source_code/worker.php"
relay: "pipes"
relay_timeout: "20s"

grpc:
listen: "tcp://0.0.0.0:9001"
proto: ${PROTO}

max_send_msg_size: 50
max_recv_msg_size: 50
max_connection_idle: 0s
max_connection_age: 0s
max_connection_age_grace: 0s
max_concurrent_streams: 10
ping_time: 1s
timeout: 200s

pool:
num_workers: ${NUM_WORKERS}
max_jobs: ${MAX_JOBS}
allocate_timeout: ${ALLOCATE_TIMEOUT}
destroy_timeout: 60

logs:
mode: development
encoding: console
level: debug
output: stderr

• lsof -i -P -n | grep LISTEN

rr 127 root 3u IPv6 25216 0t0 TCP *:6061 (LISTEN)
rr 127 root 9u IPv4 27719 0t0 TCP 127.0.0.1:2114 (LISTEN)
rr 127 root 12u IPv4 25224 0t0 TCP *:9001 (LISTEN)
rr 127 root 16u IPv4 25225 0t0 TCP 127.0.0.1:6001 (LISTEN)

• rr -c .rr.yaml workers

Workers of [grpc]:
+---------+-----------+---------+---------+---------+--------------------+
| PID | STATUS | EXECS | MEMORY | CPU% | CREATED |
+---------+-----------+---------+---------+---------+--------------------+
| 134 | ready | 0 | 55 MB | 0.04 | 2 hours ago |
| 138 | ready | 0 | 55 MB | 0.07 | 2 hours ago |
+---------+-----------+---------+---------+---------+--------------------+

• curl http://127.0.0.1:2114/health?plugin=http --> Service: http not found
• curl http://127.0.0.1:2114/health?plugin=grpc --> Service: grpc not found
• curl http://127.0.0.1:2114/health?plugin=rpc --> Service: rpc not found

[rustatian]

Hey @lanphan, thanks for the question.
Unfortunately, the grpc plugin doesn't have an implementation of the statuses (provided by the Status plugin).
It's very easy to implement them (for the workers) for the grpc plugin FR. Feel free to thumb up for this ticket 😃
For the grpc we have 2 options:

1. Native health checks: https://github.com/grpc/grpc/blob/master/doc/health-checking.md
2. Implement a status plugin interface and provide worker statistics with it.

Regarding your curl requests. You don't have an http plugin in the configuration, thus health for it returns with Service: HTTP not found.
RPC doesn't have statuses as well as the grpc as I mentioned above 😃

Also, feel free to join our discord channel https://discord.gg/TFeEmCs

[lanphan]

Thank @rustatian.
I have another question: in case I enable http plugin, and AFAIK, pool of workers is shared for all plugins (http, grpc ...)
--> therefore, I can use /health?plugin=http to have a status of free workers for grpc?

Noted that I need to have health endpoint to check the availability of workers for now, no need to have complex check like your document (grpc service available + other checks ...)

[rustatian]

Actually, the server plugin, which is allocating workers is shared (FR). But http pool is dedicated to the http plugin and not shared with others as well as grpc, tcp, jobs plugins pools.
For the grpc will be available (in the next minor release) 2 checks - native (which is described in the doc) and RR specific with the /health?plugin=grpc. Also, keep in mind, that for the k8s you need readiness checks, not healthchecks -> /ready?plugin=http. The difference between them is that the readiness check returns 200OK if there is at least 1 Ready to accept request worker.

[lanphan]

Thank @rustatian . I'm looking forward to your next minor version, so that I can use /ready?plugin=grpc directly

[lanphan]

My last question, @rustatian , when will you release your next minor version? :)

[rustatian]

My last question, @rustatian , when will you release your next minor version? :)

Mid-Feb approximately 😃

[lanphan]

Super cool, thanks so much, @rustatian

[rustatian]

You are very welcome 👍🏻

[lanphan]

Hi @rustatian ,
I see that RR have new version 2.8, and it has Check grpc method.
My question is, which method I need to use to check readiness for RR service?

• Check grpc method
• or, /ready?plugin=grpc as we discussed above?

Thanks so much for your support.

[lanphan]

For your information:

• CHECK grpc method --> works, I use grpc_health_probe utility

/usr/bin/grpc_health_probe -addr=localhost:9001
status: SERVING

• ready?plugin=grpc returns "grpc not found"

[rustatian]

For your information:

• CHECK grpc method --> works, I use grpc_health_probe utility

/usr/bin/grpc_health_probe -addr=localhost:9001
status: SERVING

• ready?plugin=grpc returns "grpc not found"

Thanks, HTTP checks for the GRPC plugin will be in the next releases.

[rustatian]

@lanphan If that is suitable for you, you may use the grpc watch and check healthchecks at the moment.

[lanphan]

hi @rustatian ,

Check is ok now, thanks so much.

[TemKaa1337]

@rustatian
Hi!
Is gRPC readiness check implemented as HTTP check now? like /ready?plugin=grpc

[TemKaa1337]

Thanks a lot for your answers and possible solution, you helped me a lot!)
Will try to implement it 😃

[rustatian]

@TemKaa1337 roadrunner-server/grpc#130
Will be in 2023.3.7 release this Thursday.

[TemKaa1337]

Wow, thanks a lot!
Will try it out and return with feedback :)

[TemKaa1337]

@rustatian
Hi!
Seems like everything is working as expected and as we want, HUGE thanks for this feature!

[rustatian]

Enjoy RR ❤️