Docker container healthchecks

[sekhavati]

I'm trying to setup health checks at the container level for my Apollo Router instances. Either via a Docker HEALTHCHECK instruction or at the ECS Task definition level (which may be the same thing ultimately), eg:

HEALTHCHECK CMD curl -f http://127.0.0.1:4000/health || exit 1

However, it appears the Apollo Router image is built with Distroless. This limits the ability to run such a command from within the container given there's no shell installed etc.

I've come to the conclusion that it won't be possible to utilise these types of healthchecks, but wanted to ask here in case there's something I've overlooked?

[o0Ignition0o]

Hey!

The healthcheck path is /.well-known/apollo/server-health by default, and can be configured as shown in the docs .

I think the -debug version of our images does include curl, which could help, but In general I would probably probe the server from outside the container?

[sekhavati]

@o0Ignition0o thanks for the response. I was aware of the healthcheck endpoint you mentioned, I've configured it to be /health in our instance (apologies I wasn't clear earlier).

In general I would probably probe the server from outside the container?

Yeah as you say, I've setup health checks from outside the container too via our load balancer. These work perfectly fine 👍

It was mainly for AWS ECS's sake. It does its own health check reports on container instances its running for you (see screenshot) using the mechanism I mentioned earlier. However it runs the command from within the container I believe and hence fails due to use of Distroless.

I wasn't aware of the -debug version, good to know, thanks! I think I'll probably swap one set of problems for another on reflection (security vulnerabilities etc), so will just omit doing health checks at the ECS level.

FYI you can reproduce what I was referring to without needing to use AWS as follows:

1. Create a Dockerfile with:

FROM ghcr.io/apollographql/router:v0.15.1
EXPOSE 4000
HEALTHCHECK CMD curl -f http://127.0.0.1:4000/health || exit 1
ENTRYPOINT ["./router"]

2. Build the image and run a container
3. Run docker ps

Result => you'll notice the Docker CLI reports the container as being in an unhealthy status

[shanesveller]

AFAIK this is something you could include directly in your ECS task definitions at deploy-time without requiring any changes to the upstream image:

https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_ContainerDefinition.html#ECS-Type-ContainerDefinition-healthCheck
https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_HealthCheck.html
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ecs-taskdefinition-healthcheck.html

Since the Dockerfile HEALTHCHECK directive isn't supported/honored by most other container orchestrators, upstreaming it may not be broadly beneficial.

[sekhavati]

Morning @shanesveller, thanks for that. Unfortunately that was the first thing I tried. I think whether you set it via the task definition or the HEALTHCHECK Docker instruction, it's the same thing ultimately. The documentation says it'll default to whatever the image declares if nothing is explicitly set at the task level.

Under the hood I presume it's trying to run the healthcheck command via something like:

$ docker exec -it <container-id> <healthcheck-cmd>

So it attempts to run <healthcheck-cmd> within the container. But this fails because of Distroless, due to there being no shell whatsoever for it to run the command 😢

Since the Dockerfile HEALTHCHECK directive isn't supported/honored by most other container orchestrators, upstreaming it may not be broadly beneficial.

Good to know, thanks 👍