Not able to pass healthcheck

[mbajur]

Hello! I have a perfectly fine working Dockerfile with production config for my app. I'm able to run it locally and everything seems to be fine with the image itself but once i use it with kamal, my healtcheck starts complaining without any helpfull output displayed.

All i can see is:

Ensure app can pass healthcheck...
  INFO [db9b8c4e] Running docker run --detach --name healthcheck-xxx-36014653139f534d2728d2656a36bc6e573c3d60 --publish 3999:3000 --label service=healthcheck-xxx -e KAMAL_CONTAINER_NAME="healthcheck-xxx" --env-file .kamal/env/roles/xxx-web.env --health-cmd "ls" --health-interval "1s" --volume /home/app/xxx-cache:/app/tmp/cache --volume /home/app/xxx-shared:/app/shared --volume /home/app/xxx-storage:/app/storage --network "private" ghcr.io/mbajur/xxx:36014653139f534d2728d2656a36bc6e573c3d60 on x.x.x.x
  INFO [db9b8c4e] Finished in 6.367 seconds with exit status 0 (successful).
  INFO [b08e73ca] Running docker container ls --all --filter name=^healthcheck-xxx-36014653139f534d2728d2656a36bc6e573c3d60$ --quiet | xargs docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}' on x.x.x.x
  INFO [b08e73ca] Finished in 1.611 seconds with exit status 0 (successful).
  INFO container not ready (unhealthy), retrying in 1s (attempt 1/7)...
  INFO [9338dbbb] Running docker container ls --all --filter name=^healthcheck-xxx-36014653139f534d2728d2656a36bc6e573c3d60$ --quiet | xargs docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}' on x.x.x.x
  INFO [9338dbbb] Finished in 0.277 seconds with exit status 0 (successful).
  INFO container not ready (unhealthy), retrying in 2s (attempt 2/7)...
  INFO [9f8534bf] Running docker container ls --all --filter name=^healthcheck-xxx-36014653139f534d2728d2656a36bc6e573c3d60$ --quiet | xargs docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}' on x.x.x.x
  INFO [9f8534bf] Finished in 0.291 seconds with exit status 0 (successful).
  INFO container not ready (unhealthy), retrying in 3s (attempt 3/7)...
  INFO [9e1cf807] Running docker container ls --all --filter name=^healthcheck-xxx-36014653139f534d2728d2656a36bc6e573c3d60$ --quiet | xargs docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}' on x.x.x.x
  INFO [9e1cf807] Finished in 0.283 seconds with exit status 0 (successful).
  INFO container not ready (unhealthy), retrying in 4s (attempt 4/7)...
  INFO [a0eec7d7] Running docker container ls --all --filter name=^healthcheck-xxx-36014653139f534d2728d2656a36bc6e573c3d60$ --quiet | xargs docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}' on x.x.x.x
  INFO [a0eec7d7] Finished in 0.282 seconds with exit status 0 (successful).
  INFO container not ready (unhealthy), retrying in 5s (attempt 5/7)...
  INFO [c3e00218] Running docker container ls --all --filter name=^healthcheck-xxx-36014653139f534d2728d2656a36bc6e573c3d60$ --quiet | xargs docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}' on [185](https://github.com/mbajur/xxx/actions/runs/8222074109/job/22483228721#step:7:186).25.150.[187](https://github.com/mbajur/xxx/actions/runs/8222074109/job/22483228721#step:7:188)
  INFO [c3e00218] Finished in 0.285 seconds with exit status 0 (successful).
  INFO container not ready (unhealthy), retrying in 6s (attempt 6/7)...
  INFO [e246f316] Running docker container ls --all --filter name=^healthcheck-xxx-36014653139f534d2728d2656a36bc6e573c3d60$ --quiet | xargs docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}' on 185.25.150.187
  INFO [e246f316] Finished in 0.281 seconds with exit status 0 (successful).
  INFO container not ready (unhealthy), retrying in 7s (attempt 7/7)...
  INFO [56bd0913] Running docker container ls --all --filter name=^healthcheck-xxx-36014653139f534d2728d2656a36bc6e573c3d60$ --quiet | xargs docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}' on 185.25.150.187
  INFO [56bd0913] Finished in 0.282 seconds with exit status 0 (successful).
  INFO [01ccc6a2] Running docker container ls --all --filter name=^healthcheck-xxx-36014653139f534d2728d2656a36bc6e573c3d60$ --quiet | xargs docker logs --tail 50 2>&1 on x.x.x.x
  INFO [01ccc6a2] Finished in 0.433 seconds with exit status 0 (successful).
 ERROR Switch to inspect mode.
 ERROR {
  "Status": "unhealthy",
  "FailingStreak": 0,
  "Log": [
  ]
}

Because the Log is empty, i'm not really able to tell what's wrong with the container itself. All i was able to see was that, during the deployment, there is a container for web being created with irb command (which is nothing i defined, my healthcheck command is bundle exec puma -C config/puma.rb so not sure where irb is coming from?).

Also, when running Dockerfile locally, i can see it produces output so i'm not sure why there is no output when kamal tries to run a healthcheck. Here is my deploy.yml file:

service: xxx

image: mbajur/xxx

volumes:
  - "/home/app/xxx-cache:/app/tmp/cache"
  - "/home/app/xxx-shared:/app/shared"
  - "/home/app/xxx-storage:/app/storage"

servers:
  web:
    hosts:
      - x.x.x.x
    labels:
      traefik.http.routers.xxx.entrypoints: websecure
      traefik.http.routers.xxx.rule: Host(`xxx.xxx.xxx`)
      traefik.http.routers.xxx.tls.certresolver: letsencrypt
    options:
      network: "private"

registry:
  server: ghcr.io
  username: mbajur

  password:
    - KAMAL_REGISTRY_PASSWORD

# Inject ENV variables into containers (secrets come from .env).
# Remember to run `kamal env push` after making changes!
env:
  clear:
    HOSTNAME: xxx.xxx.xxx
    DB_HOST: x.x.x.x
    RAILS_SERVE_STATIC_FILES: true
    RAILS_LOG_TO_STDOUT: true
  secret:
    - POSTGRES_PASSWORD
    - RAILS_MASTER_KEY

ssh:
  user: app

builder:
  dockerfile: Dockerfile.production
  multiarch: false
  cache:
    type: registry

accessories:
  db:
    image: postgres:15
    host: x.x.x.x
    port: 5432
    env:
      clear:
        POSTGRES_USER: "xxx"
        POSTGRES_DB: 'xxx_production'
      secret:
        - POSTGRES_PASSWORD
    files:
      - config/init.sql:/docker-entrypoint-initdb.d/setup.sql
    directories:
      - data:/var/lib/postgresql/data
    options:
      network: "private"

traefik:
  args:
    accesslog: true
  options:
    network: "private"
    publish:
      - "443:443"
    volume:
      - "/letsencrypt/acme.json:/letsencrypt/acme.json"
  args:
    entryPoints.web.address: ":80"
    entryPoints.websecure.address: ":443"
    entryPoints.web.http.redirections.entryPoint.to: websecure # We want to force https
    entryPoints.web.http.redirections.entryPoint.scheme: https
    entryPoints.web.http.redirections.entrypoint.permanent: true
    certificatesResolvers.letsencrypt.acme.email: "[email protected]"
    certificatesResolvers.letsencrypt.acme.storage: "/letsencrypt/acme.json" # Must match the path in `volume`
    certificatesResolvers.letsencrypt.acme.httpchallenge: true
    certificatesResolvers.letsencrypt.acme.httpchallenge.entrypoint: web

healthcheck:
  path: /health/ready
  port: 4000
  cmd: bundle exec puma -C config/puma.rb

Please let me know if there is anything i can do to help you help me or if there is anything obvious i'm missing in here? :)

[sebastienbourda]

I have same issue. Did you find a solution ?

[RomanKrasavtsev]

Try to increase value for max_attempts or interval https://kamal-deploy.org/docs/configuration/healthchecks/

[intrip]

Why you are using bundle exec puma as health check instead of checking the health of the app via curl?

You can ssh in the machine and see what's going on in the Healthcheck container via docker logs -f ID

Also if you are stuck you can disable the health check, but you'll lose the zero-downtime deployment feature.

[jmcbri]

Thanks. How do you disable the healthcheck? Something like:

healthcheck: false

[intrip]

You can use:

healthcheck:
  cmd: true

[jmcbri]

Thanks @intrip ! In the above, do disable it would be cmd: true Or would it be cmd: false ?

[imWildCat]

did you add this to your routes.rb if it's rails?

  # Reveal health status on /up that returns 200 if the app boots with no exceptions, otherwise 500.
  # Can be used by load balancers and uptime monitors to verify that the app is live.
  get "up" => "rails/health#show", as: :rails_health_check