Readiness probe failing when running container locally

[sumitd2]

Hi team,

I am running the k8ssandra/cass-management-api:4.1.2 container locally on my VM, and then doing a curl for the readiness probe which fails. The liveness probe passes. What am I missing?

Container Logs:

root@sumit-osa4-isv:~# docker run --privileged -e USE_MGMT_API=true -p 8080:8080 -it --rm k8ssandra/cass-management-api:4.1.2
Starting Management API
Default Java version: 11.0.20
Running java -Xms128m -Xmx128m -jar /opt/management-api/datastax-mgmtapi-server.jar --cassandra-socket /tmp/cassandra.sock --host tcp://0.0.0.0:8080 --host file:///tmp/oss-mgmt.sock --cassandra-home /opt/cassandra
INFO  [main] 2025-01-02 09:23:35,661 Cli.java:727 - Found Cassandra binary on PATH: /opt/cassandra/bin/cassandra
INFO  [main] 2025-01-02 09:23:36,284 Cli.java:376 - Cassandra Version 4.1.2
INFO  [main] 2025-01-02 09:23:36,600 ResteasyDeploymentImpl.java:612 - RESTEASY002225: Deploying jakarta.ws.rs.core.Application: class com.datastax.mgmtapi.ManagementApplication
INFO  [main] 2025-01-02 09:23:36,604 ResteasyDeploymentImpl.java:635 - RESTEASY002220: Adding singleton resource com.datastax.mgmtapi.resources.LifecycleResources from Application class com.datastax.mgmtapi.ManagementApplication
INFO  [main] 2025-01-02 09:23:36,604 ResteasyDeploymentImpl.java:635 - RESTEASY002220: Adding singleton resource com.datastax.mgmtapi.resources.K8OperatorResources from Application class com.datastax.mgmtapi.ManagementApplication
INFO  [main] 2025-01-02 09:23:36,604 ResteasyDeploymentImpl.java:635 - RESTEASY002220: Adding singleton resource com.datastax.mgmtapi.resources.KeyspaceOpsResources from Application class com.datastax.mgmtapi.ManagementApplication
INFO  [main] 2025-01-02 09:23:36,604 ResteasyDeploymentImpl.java:635 - RESTEASY002220: Adding singleton resource com.datastax.mgmtapi.resources.v1.KeyspaceOpsResources from Application class com.datastax.mgmtapi.ManagementApplication
INFO  [main] 2025-01-02 09:23:36,605 ResteasyDeploymentImpl.java:635 - RESTEASY002220: Adding singleton resource com.datastax.mgmtapi.resources.MetadataResources from Application class com.datastax.mgmtapi.ManagementApplication
INFO  [main] 2025-01-02 09:23:36,605 ResteasyDeploymentImpl.java:635 - RESTEASY002220: Adding singleton resource com.datastax.mgmtapi.resources.NodeOpsResources from Application class com.datastax.mgmtapi.ManagementApplication
INFO  [main] 2025-01-02 09:23:36,605 ResteasyDeploymentImpl.java:635 - RESTEASY002220: Adding singleton resource com.datastax.mgmtapi.resources.v1.NodeOpsResources from Application class com.datastax.mgmtapi.ManagementApplication
INFO  [main] 2025-01-02 09:23:36,606 ResteasyDeploymentImpl.java:635 - RESTEASY002220: Adding singleton resource com.datastax.mgmtapi.resources.TableOpsResources from Application class com.datastax.mgmtapi.ManagementApplication
INFO  [main] 2025-01-02 09:23:36,606 ResteasyDeploymentImpl.java:635 - RESTEASY002220: Adding singleton resource com.datastax.mgmtapi.resources.v1.TableOpsResources from Application class com.datastax.mgmtapi.ManagementApplication
INFO  [main] 2025-01-02 09:23:36,606 ResteasyDeploymentImpl.java:635 - RESTEASY002220: Adding singleton resource com.datastax.mgmtapi.resources.AuthResources from Application class com.datastax.mgmtapi.ManagementApplication
INFO  [main] 2025-01-02 09:23:36,606 ResteasyDeploymentImpl.java:635 - RESTEASY002220: Adding singleton resource com.datastax.mgmtapi.resources.v2.RepairResourcesV2 from Application class com.datastax.mgmtapi.ManagementApplication
INFO  [main] 2025-01-02 09:23:36,606 ResteasyDeploymentImpl.java:635 - RESTEASY002220: Adding singleton resource com.datastax.mgmtapi.resources.v2.TokenResourcesV2 from Application class com.datastax.mgmtapi.ManagementApplication
INFO  [main] 2025-01-02 09:23:36,607 ResteasyDeploymentImpl.java:635 - RESTEASY002220: Adding singleton resource io.swagger.v3.jaxrs2.integration.resources.OpenApiResource from Application class com.datastax.mgmtapi.ManagementApplication
INFO  [main] 2025-01-02 09:23:36,607 ResteasyDeploymentImpl.java:643 - RESTEASY002210: Adding provider singleton io.swagger.v3.jaxrs2.SwaggerSerializers from Application class com.datastax.mgmtapi.ManagementApplication
Started service on tcp://0.0.0.0:8080
INFO  [main] 2025-01-02 09:23:36,824 ResteasyDeploymentImpl.java:612 - RESTEASY002225: Deploying jakarta.ws.rs.core.Application: class com.datastax.mgmtapi.ManagementApplication
INFO  [main] 2025-01-02 09:23:36,824 ResteasyDeploymentImpl.java:635 - RESTEASY002220: Adding singleton resource com.datastax.mgmtapi.resources.LifecycleResources from Application class com.datastax.mgmtapi.ManagementApplication
INFO  [main] 2025-01-02 09:23:36,824 ResteasyDeploymentImpl.java:635 - RESTEASY002220: Adding singleton resource com.datastax.mgmtapi.resources.K8OperatorResources from Application class com.datastax.mgmtapi.ManagementApplication
INFO  [main] 2025-01-02 09:23:36,824 ResteasyDeploymentImpl.java:635 - RESTEASY002220: Adding singleton resource com.datastax.mgmtapi.resources.KeyspaceOpsResources from Application class com.datastax.mgmtapi.ManagementApplication
INFO  [main] 2025-01-02 09:23:36,825 ResteasyDeploymentImpl.java:635 - RESTEASY002220: Adding singleton resource com.datastax.mgmtapi.resources.v1.KeyspaceOpsResources from Application class com.datastax.mgmtapi.ManagementApplication
INFO  [main] 2025-01-02 09:23:36,825 ResteasyDeploymentImpl.java:635 - RESTEASY002220: Adding singleton resource com.datastax.mgmtapi.resources.MetadataResources from Application class com.datastax.mgmtapi.ManagementApplication
INFO  [main] 2025-01-02 09:23:36,825 ResteasyDeploymentImpl.java:635 - RESTEASY002220: Adding singleton resource com.datastax.mgmtapi.resources.NodeOpsResources from Application class com.datastax.mgmtapi.ManagementApplication
INFO  [main] 2025-01-02 09:23:36,825 ResteasyDeploymentImpl.java:635 - RESTEASY002220: Adding singleton resource com.datastax.mgmtapi.resources.v1.NodeOpsResources from Application class com.datastax.mgmtapi.ManagementApplication
INFO  [main] 2025-01-02 09:23:36,825 ResteasyDeploymentImpl.java:635 - RESTEASY002220: Adding singleton resource com.datastax.mgmtapi.resources.TableOpsResources from Application class com.datastax.mgmtapi.ManagementApplication
INFO  [main] 2025-01-02 09:23:36,825 ResteasyDeploymentImpl.java:635 - RESTEASY002220: Adding singleton resource com.datastax.mgmtapi.resources.v1.TableOpsResources from Application class com.datastax.mgmtapi.ManagementApplication
INFO  [main] 2025-01-02 09:23:36,826 ResteasyDeploymentImpl.java:635 - RESTEASY002220: Adding singleton resource com.datastax.mgmtapi.resources.AuthResources from Application class com.datastax.mgmtapi.ManagementApplication
INFO  [main] 2025-01-02 09:23:36,826 ResteasyDeploymentImpl.java:635 - RESTEASY002220: Adding singleton resource com.datastax.mgmtapi.resources.v2.RepairResourcesV2 from Application class com.datastax.mgmtapi.ManagementApplication
INFO  [main] 2025-01-02 09:23:36,826 ResteasyDeploymentImpl.java:635 - RESTEASY002220: Adding singleton resource com.datastax.mgmtapi.resources.v2.TokenResourcesV2 from Application class com.datastax.mgmtapi.ManagementApplication
INFO  [main] 2025-01-02 09:23:36,826 ResteasyDeploymentImpl.java:635 - RESTEASY002220: Adding singleton resource io.swagger.v3.jaxrs2.integration.resources.OpenApiResource from Application class com.datastax.mgmtapi.ManagementApplication
INFO  [main] 2025-01-02 09:23:36,826 ResteasyDeploymentImpl.java:643 - RESTEASY002210: Adding provider singleton io.swagger.v3.jaxrs2.SwaggerSerializers from Application class com.datastax.mgmtapi.ManagementApplication
INFO  [main] 2025-01-02 09:23:36,853 IPCController.java:111 - Starting Server
INFO  [main] 2025-01-02 09:23:36,858 IPCController.java:121 - Started Server
Started service on file:///tmp/oss-mgmt.sock
INFO  [pool-2-thread-1] 2025-01-02 09:23:46,982 LifecycleResources.java:225 - Started Cassandra
INFO  [nioEventLoopGroup-2-2] 2025-01-02 09:23:55,619 Cli.java:672 - address=/172.17.0.1:38598 url=/api/v0/probes/readiness status=500 Internal Server Error
INFO  [nioEventLoopGroup-2-1] 2025-01-02 09:24:29,364 Cli.java:672 - address=/172.17.0.1:47086 url=/api/v0/probes/liveliness status=404 Not Found
INFO  [nioEventLoopGroup-2-2] 2025-01-02 09:24:42,022 Cli.java:672 - address=/172.17.0.1:51198 url=/api/v0/probes/liveness status=200 OK

Curl logs:

root@sumit-osa4-isv:~# curl http://127.0.0.1:8080/api/v0/probes/readiness
Internal connection to Cassandra closed
root@sumit-osa4-isv:~# curl http://127.0.0.1:8080/api/v0/probes/liveness
OK

Thanks in advance.

┆Issue is synchronized with this Jira Story by Unito
┆Issue Number: MAPI-90

[emerkle826]

@sumitd2 Assuming your environment has enough resources to run the image, the readiness probe can sometimes take a bit of time to respond. Cassandra has to fully come up and be responsive to a CQL query for the readiness probe to respond with a 200 OK. Until then, it will return a 500 error.

If you have given the server some time to come up and the readiness probe is still not returning a 200 OK, you may need to investigate the Cassandra logs in the container to see if there are some indications as to why the server is not able to come up. You can look at the system.log maybe and see if anything stands out:

docker exec -it <container_name> sh -c "tail -100 /var/log/cassandra/system.log"

One thing you may notice if you do this on your container running Cassandra 4.1.2 (with the command you provided) is that there will be an exception complaining about MCAC. As of Cassandra 4.1.x (and newer), MCAC is no longer supported. There is a built-in metrics collector in Management API that replaces it, so you will need to disable MCAC. You can do so with MGMT_API_DISABLE_MCAC=true, so change your startup to this:

docker run --privileged -e USE_MGMT_API=true -e MGMT_API_DISABLE_MCAC=true -p 8080:8080 -it --rm k8ssandra/cass-management-api:4.1.2

This MCAC error shouldn't prevent it from running currently however.